<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hamza Belghiti on Medium]]></title>
        <description><![CDATA[Stories by Hamza Belghiti on Medium]]></description>
        <link>https://medium.com/@hamza.lbelghiti?source=rss-887d95bc7820------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5ijU6EaZ9EQvL2M5VIcVFQ.png</url>
            <title>Stories by Hamza Belghiti on Medium</title>
            <link>https://medium.com/@hamza.lbelghiti?source=rss-887d95bc7820------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:05:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hamza.lbelghiti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data-Driven Decisions Using the Modern Data Stack]]></title>
            <link>https://medium.com/@hamza.lbelghiti/data-driven-decisions-using-the-modern-data-stack-999f901da0f5?source=rss-887d95bc7820------2</link>
            <guid isPermaLink="false">https://medium.com/p/999f901da0f5</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[Hamza Belghiti]]></dc:creator>
            <pubDate>Fri, 24 May 2024 00:48:57 GMT</pubDate>
            <atom:updated>2026-06-16T13:32:00.260Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOZvyZjzsJoeXRgZy0adWQ.png" /></figure><p>During my current end-of-studies internship at <a href="https://blent.ai/">Blent.ai</a> as an Analytics Engineer Intern, I had the opportunity to work on various data engineering and analytics tasks. This included working on data integration, data transformation, modeling data, creating KPIs, and developing dashboards. In this blog post, I want to <strong>share my experience working with some of the modern data stack tools</strong> and how they made my day-to-day data engineering and analytics easy and impactful.</p><h3><strong>But first: </strong>What is a Modern Data Stack?</h3><p>The Modern Data Stack is a <strong>collection of tools and technologies, hosted on the cloud or on-premise, aimed at collecting, processing, storing, analyzing, and visualizing data.</strong> The goal of the tools in a modern data stack is to handle large volumes of data, support real-time analytics, and enable data-driven decision-making.</p><p>The modern data stack is also the consequence of a shift in the design of the data pipelines paradigm from <strong>Extract Transform Load (ETL)</strong> to<strong> Extract Load Transform (ELT)</strong>. This means data is first loaded into a warehouse and then transformed, offering greater flexibility and scalability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*GAwNOM-wM-rJ6bOcENX5GQ.png" /><figcaption>Modern Data Stack Architecture (Credit: <a href="https://blog.skyvia.com/">https://blog.skyvia.com/</a>)</figcaption></figure><p><strong>Key Differences Between Classical ETL and Modern ELT:</strong></p><ul><li><strong>ETL: </strong>Data is extracted from different sources, transformed to meet analysis needs, and then loaded into a data warehouse. This can be time-consuming and less adaptable to changes.</li><li><strong>ELT</strong>: Data is extracted and directly loaded into the data warehouse in its raw form. Transformation occurs afterward, leveraging the power of the modern cloud-based data warehouse’s processing capabilities. This approach is more agile and scalable, accommodating the growing volume and variety of data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hQ94XXC3u4uFMxUY.png" /><figcaption>ETL vs ELT (Credit: <a href="https://www.striim.com/">https://www.striim.com/</a>)</figcaption></figure><p>The Modern Data Stack tools can be grouped into different categories, <strong>each corresponding to a specific aspect of the data workflow.</strong> For example, there are tools responsible for data integration while others allow you to create a data catalog from a database. Each can operate independently or interact with others to communicate and share data.</p><p>Companies can combine different tools for different tasks, creating <strong>THEIR OWN Data Stack</strong>. You can take a look at the different data stacks used by popular tech companies like Airbnb, Amazon, Apple etc. on this <a href="https://www.moderndatastack.xyz/stacks">website</a>.</p><p>A typical modern data stack consists of the following:</p><h4>1. Data Integration</h4><p>Companies often collect data from different sources, such as internal databases like PostgreSQL, MongoDB, CRM (Hubspot, Salesforce), ERP (SAP) platforms, or external applications such as APIs, web applications, payment tools, etc.</p><p>To integrate data from various sources, tools like <a href="https://airbyte.com/"><strong>Airbyte</strong></a><strong>, Fivetran, and Stitch </strong>are invaluable. These tools facilitate <strong>seamless data extraction (sources) and loading into data warehouses (destinations)</strong>, supporting numerous data sources and providing automated, incremental data updates.</p><blockquote>The data is loaded in its raw format into the Data Warehouse! Transformations are done later on.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p2iWh0ConzFFzNGA.png" /><figcaption>Example of Data Integration tool: Airbyte</figcaption></figure><h4>2. Data Storage</h4><p>The collected data is then stored in a cloud-based data warehouse. There are mainly four different storage methods.</p><ul><li><strong>Data warehouse</strong> stores structured data for analytics. Options like <a href="https://cloud.google.com/bigquery"><strong>BigQuery</strong></a><strong> in GCP, Snowflake, Redshift in AWS, and Synapse </strong>in Azure provide scalable storage and powerful processing capabilities. These platforms allow for efficient querying and analysis of large datasets using SQL queries.</li><li><strong>Data lake </strong>is<strong> </strong>a centralized, highly flexible storage repository that stores large amounts of structured and unstructured, in the form of a CSV or JSON file for example, data in its raw, original, and unformatted form and is often used for advanced analytics. Options for data lakes include <strong>GCS on GCP, AWS S3 Buckets, and Azure Data Lake Storage</strong>.</li><li><strong>Data lakehouse </strong>is a new, big-data storage architecture that combines the best features of both data warehouses and data lakes. A data lakehouse enables a single repository for all your data (structured, semi-structured, and unstructured). It enables ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data. A popular option is <a href="https://www.databricks.com/"><strong>Databricks</strong></a><strong>.</strong></li><li><strong>Data mesh</strong> is an organizational and technical approach that decentralizes data ownership among domain data owners. The idea behind data mesh architecture is to organize data based on business processes like sales, marketing, finance, and more. <a href="https://cloud.google.com/dataplex/docs/build-a-data-mesh"><strong>Dataplex</strong></a><strong> </strong>is a GCP solution to implement a data mesh architecture.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gni-CLhDeftjwRTx" /><figcaption>Data Warehouse vs Data Lake vs Data Lakehouse</figcaption></figure><h4>3. Data Modeling and Transformation</h4><p>Raw data collected from different sources must be transformed, converted, cleaned, and structured into a usable format that can be used for analysis and support decision-making processes. For example, to analyze a hotel’s sales across different channels, you need to match data from its website, AirBnb or Booking. This data is not presented in the same way and it is therefore necessary to reprocess it to <strong>obtain uniform information</strong> on sales figures by channel and by day.</p><p>For data modeling and transformation, <a href="https://www.getdbt.com/product/dbt-cloud"><strong>dbt</strong></a><strong> (Data Build Tool)</strong> stands out as a popular choice. It enables analysts to transform data within the warehouse using SQL-based transformations. <strong>Dataform is another option that very </strong>integrates well with BigQuery, while <strong>Spark</strong> on Databricks offers distributed data processing for complex transformations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G1UZ19eOUBsBl6kn" /><figcaption>dbt workflow</figcaption></figure><h4>4. Workflow Orchestration</h4><p>When we design a data pipeline, we follow a series of steps, beginning with collecting and integrating the data, to storing and transforming the data. A data pipeline <strong>is designed to move data from source to destination.</strong> These pipelines are scheduled and automated by an orchestrator.</p><p>Plenty of orchestration tools exist, the most popular and the most used is <a href="https://airflow.apache.org/"><strong>Airflow</strong></a><strong>.</strong> Many other tools have emerged since then inspired by Airflow, like <strong>Mage, Prefect, and Dagster</strong>. These tools allow for the scheduling, monitoring, and management of data pipelines, ensuring that data processes run smoothly and reliably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7ljQdT5CNWKJyTq2.png" /><figcaption>Example of a Mage data pipeline</figcaption></figure><h4>5. Business Intelligence</h4><p>Business Intelligence refers to the use of data-related tools, methods, and best practices to make intelligent use of company data and <strong>draw valuable insights </strong>that can <strong>help the business decision-making process.</strong></p><p>Business intelligence tools are essential in the Data Stack of every company.<strong> </strong><a href="https://lookerstudio.google.com/u/0/"><strong>Looker Studio</strong></a><strong> </strong>integrates seamlessly with BigQuery, while <strong>Metabase, Superset, PowerBI, and Tableau</strong> also have good visualization and reporting capabilities. These tools help turn raw data into actionable insights through interactive dashboards and reports. Whether you are a technical or non-technical person, BI tools allow you to explore and interact with data through a charting interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OzhZ-74KccoAp-a5.png" /><figcaption>Example of a Looker Studio Dashboard</figcaption></figure><h4>6. Data Privacy and Governance</h4><p>Ensuring data privacy and governance is crucial for a company to maintain control over its data and know who is accessing what data. Data governance<strong> is a data management concept.</strong> It is a measure of the<strong> control a company has over its data</strong>. This control can be achieved through high-quality data, visibility on data pipelines, actionable rights management, and clear accountability. Data governance encompasses the people, processes, and tools required to create consistent and proper handling of a company’s data.</p><p>Tools like <a href="https://open-metadata.org/"><strong>Open Metadata</strong></a>, <strong>Datahub, and Amundsen</strong> provide frameworks for managing metadata, ensuring data quality, and maintaining compliance with data regulations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aP9azPv61bwmzeet" /><figcaption>Example of a use case of Open Metadata</figcaption></figure><h4>7. Reverse ETL</h4><p>Sometimes companies need the newly cleaned and transformed data sitting in their data warehouse back into their CRM tools. Imagine you have data about your customers from different sources: Hubspot, MongoDB, Salesforce etc. The incoming data is inconsistent, so you had to load them to your Data Warehouse, clean them, and unify them<em> (also known as the </em><strong><em>Customer Data Platform (CDP)</em></strong><em>)</em> for your analytical team. But now your marketing team requests to have that unified data about the Customers in their Marketing tool, Mailchimp for example, to conduct targeted marketing campaigns. <strong>You will need to use a Reverse ETL pipeline.</strong></p><p>Reverse ETL enables companies to <strong>move transformed data from their cloud warehouse out into operational business tools</strong>. It’s a new approach to making data actionable and solving the “last mile” problem in analytics by empowering business teams to access — and act on — transformed data directly in the SaaS tools they already use every day.</p><p>For this, you can use tools like Octolus, <a href="https://hightouch.com/">Hightouch</a>, or Census. Alternatively, you can leverage data warehouse APIs and third-party tool APIs, integrating them into your workflow orchestration tools for customized data flows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YWDcdrSUTL-F7zeY" /><figcaption>How an Reverse ETL works</figcaption></figure><h4>8. Machine Learning</h4><p>The modern data stack isn’t just about data management, it also integrates seamlessly with Machine Learning (ML) tools for powerful analytics. Here are some popular cloud-based solutions: <strong>SparkML for Databricks, BigQuery ML and VertexAI (GCP), Amazon SageMaker (AWS), Azure Machine Learning (Azure).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jdd8oH4qYauIUZnH.png" /><figcaption>A Use Case of VertexAI to make an ML application</figcaption></figure><p>There are more categories of tools used in the modern data stack, but I can’t list them all, feel free to take a look at them on the <a href="https://www.moderndatastack.xyz/categories"><strong>moderndatastack</strong></a> website.</p><p><strong>😵 Feeling overwhelmed by all these tools?</strong><br><br>That’s exactly the problem I built <a href="https://craftcourse.app/">CraftCourse</a> to solve. Instead of opening 10 YouTube tabs and never finishing any of them, you describe what you want to learn say, <strong>“dbt for data transformation”</strong> or <strong>“Airflow from scratch”</strong> and AI generates a full structured learning path with modules, lessons, and a final project tailored to your level.<br><br>Sage, your personal AI tutor, has full context of your content and progress and can answer any question while you learn. Quizzes, flashcards, and study guides are generated on demand to make sure it actually sticks.</p><blockquote>👉 <a href="https://craftcourse.app/">Try it free — first 1,000 users get lifetime access to the free plan</a></blockquote><h3><strong>The Power of Modern Data Stacks</strong></h3><p>The modern data stack offers a multitude of benefits.</p><p>First of all, it’s less expensive than traditional data stacks since <strong>cloud-based solutions follow a pay-as-you-go model. </strong>The cloud-based nature of the modern data stack offers exceptional scalability. As your data volume grows, the infrastructure scales effortlessly to accommodate the increased workload. This eliminates the need to constantly invest in and manage on-premise hardware, allowing your data operations to adapt seamlessly to your business’s evolving needs.</p><p>Second is their ability to process a wide variety of data types. Unlike traditional solutions often limited to structured data, modern data stacks <strong>easily adapt to more complex formats</strong>, thus allowing for more comprehensive exploitation of available information. The support of these different data types by modern data stacks offers several advantages like a holistic view of the business, deeper analyses, and innovation because having access to a wider variety of data enables businesses to develop new products, services, and strategies based on unprecedented insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6hws0iIoEqSoINs6" /><figcaption>Customer Data Platform based on Modern Data Stack tools</figcaption></figure><h3>Who is the Modern Data Stack for?</h3><p>The modern data stack is designed to empower a wide range of users. It allows data engineers to build scalable and robust data pipelines, data analysts to explore and transform data for insights, and business users to access and visualize data for informed decision-making. By offering user-friendly tools and fostering collaboration, the modern data stack makes the power of data accessible to everyone who can benefit from it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SliSKSt2gqzeR7RA.png" /><figcaption>The future of the Modern Data Stack (Credit: Jelvix.com)</figcaption></figure><h3>Conclusion</h3><p>The modern data stack has revolutionized how we leverage data. This suite of cloud-based tools empowers businesses of all sizes to collect, store, transform, and analyze data efficiently. Using these tools together in a harmonious way will unlock the true potential of your data with a minimum cost and effort.</p><p>The benefits are undeniable: faster insights, data-driven decision-making, and the ability to handle diverse data types for a holistic view of the business. Whether you’re a data engineer building pipelines, an analyst uncovering insights, or a business user exploring trends, the modern data stack equips you with the tools you need to succeed in today’s data-driven world.</p><p>🔮<strong> BONUS:</strong></p><p>If you are looking to up-skill in the data world, here are some YouTube and LinkedIn profiles worth following:</p><ul><li><a href="https://www.youtube.com/@SeattleDataGuy">https://www.youtube.com/@SeattleDataGuy</a></li><li><a href="https://www.youtube.com/@nullQueries">https://www.youtube.com/@nullQueries</a></li><li><a href="https://www.linkedin.com/in/eczachly/">https://www.linkedin.com/in/eczachly/</a></li><li><a href="https://www.linkedin.com/in/axel-thevenot/">https://www.linkedin.com/in/axel-thevenot/</a></li></ul><p>Thank you for sticking until the end! 🙌🏻</p><p><strong>🎓 Ready to actually learn the Modern Data Stack?</strong><br><br>Reading about tools is one thing. Building real skills with them is another.</p><p>I built <a href="https://craftcourse.app/">CraftCourse</a> for exactly this: tell the AI you want to master Airflow, dbt, BigQuery, or any other tool in the stack, and it maps you a full learning path — from core basics to advanced concepts — then keeps you accountable until you finish. No more half-watched tutorials.</p><p>✅ AI-generated course outline tailored to your level <br>✅ Fill lessons with YouTube videos or any content you want <br>✅ Quizzes, notes, flashcards while you learn <br>✅ Sage AI tutor with full context of your progress <br>✅ XP, streaks, and a certificate when you finish</p><blockquote><a href="https://craftcourse.app/">Start your first course free → craftcourse.app</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=999f901da0f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Ways to Gain Real-World Project Experience as Data Students]]></title>
            <link>https://medium.com/@hamza.lbelghiti/5-ways-to-gain-real-world-project-experience-as-data-students-02260745837e?source=rss-887d95bc7820------2</link>
            <guid isPermaLink="false">https://medium.com/p/02260745837e</guid>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[jobs]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Hamza Belghiti]]></dc:creator>
            <pubDate>Fri, 27 Oct 2023 19:53:39 GMT</pubDate>
            <atom:updated>2024-08-14T22:22:50.417Z</atom:updated>
            <content:encoded><![CDATA[<h3>5 Ways to Gain Real-World Project Experience as Data Student</h3><h4>Transforming Theory into Real-world Expertise with These Platforms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUczkx_UuH1VIBn8_XV_QQ.png" /><figcaption>5 Ways to Get Real-World Project Experiences</figcaption></figure><p>As a data student, you’ve likely encountered the challenge of gaining real-world experience to complement your theoretical knowledge. While coursework and textbooks offer a strong foundation, the real magic happens when you apply your skills to practical, industry-specific projects. In this article, we’ll explore five platforms that provide data students or self-taught people in general with opportunities to gain hands-on experience and develop a portfolio that will impress future employers.</p><blockquote>This a continuation of the discussion I had with <a href="https://www.linkedin.com/in/mghoulami/"><em>Marouane Ghoulami</em></a> on his podcast on his channel <a href="https://www.youtube.com/watch?v=fit7w2jb3D4"><strong>Den Gen XYZ</strong></a><strong>.</strong></blockquote><p>Before we explore these platforms, it’s essential to lay the foundation, and for that, you’ll need to build a compelling portfolio. A strong portfolio, filled with relevant projects in your chosen field, be it data science, data analysis, or data engineering, is your ticket to success. To help you get started, I’ve gathered some invaluable resources that will guide you to create impactful and high-quality projects in these domains that will help afterward when you want to apply to the platforms we’ll discuss.</p><h4>💻 Data Science Projects</h4><ul><li>Here’s a good <a href="https://www.youtube.com/watch?v=MpF9HENQjDo&amp;list=PL2zq7klxX5ASFejJj80ob9ZAnBHdz5O1t&amp;ab_channel=KenJee">data science project</a> from scratch by <a href="https://medium.com/u/6ee1f7466557">Ken Jee</a> where you’ll learn how to develop a data science salary estimator tool, scraping job descriptions from Glassdoor, engineering features, optimizing models, and creating a user-friendly API for salary negotiation.</li><li>Another great project is one of the 12 projects present in this <a href="https://www.youtube.com/watch?v=JwSS70SZdyM&amp;ab_channel=freeCodeCamp.org">3-hour-long video</a> by <a href="https://medium.com/u/f94b47c3cfca">Chanin Nantasenamat</a> where you’ll learn how to build interactive and data-driven web apps in Python using the Streamlit library.</li></ul><h4>📊 Data Analyst Projects:</h4><ul><li>This a <a href="https://www.youtube.com/watch?v=tT4V7zguCnc&amp;ab_channel=codebasics">very insightful project </a>on how you should deal with a real-world data analysis project by <a href="https://medium.com/u/eb03d8d5a4f6">Codebasics</a> where you’ll understand how data analysts build dashboards in real-time in this video featuring an industry expert &amp; a data analytics expert.</li><li>Here’s a <a href="https://www.youtube.com/watch?v=qfyynHBFOsM&amp;list=PLUaB-1hjhk8H48Pj32z4GZgGWyylqv85f&amp;ab_channel=AlexTheAnalyst">playlist of projects</a> from <strong>Alex the Analyst </strong>where you’ll have a chance to learn about data analyst skills like data cleaning, exploration, and tools like SQL, Tableau, PowerBI, and Python.</li></ul><h4>⚙️ Data Engineering Projects:</h4><ul><li>This <a href="https://www.youtube.com/watch?v=mECDWTYiKp4&amp;list=PLrG_BXEk3kXx6KE4nBmhf6QwSHMbznP2W&amp;ab_channel=Mr.KTalksTech">data engineering project</a> by <strong>Mr. K Talks Tech </strong>is an end-to-end Azure project from data ingestion, data transformation, data loading, and reporting.</li><li><a href="https://medium.com/u/27ed83189c91">Darshil Parmar</a> did a really <a href="https://www.youtube.com/watch?v=WpQECq5Hx9g&amp;ab_channel=DarshilParmar">good project</a> where you will analyze Uber data using various tools and technologies, including GCP Storage, Python, Compute Instance, Mage Data Pipeline Tool, BigQuery, and Looker Studio.</li><li>The <strong>CodeWithYou </strong>channel has also <a href="https://www.youtube.com/watch?v=LSlt6iVI_9Y&amp;list=PL_Ct8Cox2p8UlTfHyJc3RDGuGktPNs9Q3&amp;ab_channel=CodeWithYu">great data engineering projects</a>.</li></ul><p>Okay now that you have a strong portfolio with relevant projects, we’ll explore two remarkable platforms that offer hands-on experiences, Forage and Omdena. These platforms not only provide opportunities to work on real-world projects but also help you to create a portfolio that sets you apart in the competitive landscape of data science, data engineering, and data analysis.</p><ol><li><a href="https://www.theforage.com/"><strong>The Forage</strong></a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZPKKB2qsv8iCINSZyLhAUQ.png" /><figcaption>The Forage job simulations in the “Data” field</figcaption></figure><p>Forage links aspiring IT professionals to real-world industry needs. It offers free job simulations designed by global leaders like J.P. Morgan, Walmart, and RedBull. These hands-on simulations cover various fields, such as software engineering, data science, marketing, and consulting. They provide practical experience and insights, allowing you to explore a company’s work environment. Plus, Forage’s resources are free, self-paced, and open-access, making it the perfect platform for data students to gain real-world experience and enhance their portfolios for future careers.</p><p>When you sign up, head to the <a href="https://www.theforage.com/course-catalog"><strong>Explore Job Simulations</strong></a> page. There, you can filter by your chosen domain. Once you enroll, a team member like a project manager briefs you on the tasks you’ll need to perform. Then, you work on those tasks, submit your work, and you’ll have the chance to compare it with the pro version from the company.</p><p>Once you finish the job simulation you can get your certificate of completion and you can add it to your LinkedIn or resume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiKzzvlWLuIDZBHm2iO5Sg.png" /><figcaption>Finished Data Science Job Simulation of BCG</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwh-i3qeQxT6GjC7SgoKnQ.png" /><figcaption>Certificate of Completion</figcaption></figure><p><a href="https://omdena.com/"><strong>2. Omdena</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PyBRaKLflkv8q0rlg1SstA.png" /><figcaption>Omdena Projects page</figcaption></figure><p>Omdena is a platform helping mission-driven organizations and startups build impactful AI solutions through global collaboration and empowering AI engineers worldwide to become change makers. Over 20000 AI Engineers from 120+ countries have been part of Omdena projects, building impact-driven solutions for 220+ organizations. Omdena also has local chapters in 195+ locations across 80 countries. Omdena rewards top performers with paid projects, mentorship, and free access to courses and certificates.</p><p>To begin, create an account and fill in your educational background, skills, and an English resume showcasing your relevant skill sets. Once your profile is set, visit the <a href="https://omdena.com/projects/"><strong>Projects</strong></a> page and pick 2 to 3 projects that pique your interest. Complete a brief form explaining why you’re drawn to each project and your motivation. You’ll receive an acceptance notification via email 1 to 2 weeks before the start date of the project. After that, join the project’s Slack channel and attend an introductory meeting. Following this, you’ll choose a task to work on. Upon project completion, your team will create a report and demo to be validated by the Omdena team, earning you a certificate of completion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*-BPTFWLk_D0fn6chdrCwaw.png" /><figcaption>Certificate of completion</figcaption></figure><p>Now, let’s explore two great platforms that let you take part in cool paid AI hackathons. Once you pick the hackathon you like, you can either join a team or bring your own. Once you’ve figured out what you want to build for the hackathon, choose the role you want to play in the team; data science, engineering, or analyst. These platforms, Lablab AI and Devpost are where you dive into the fun world of AI innovation.</p><p><a href="https://lablab.ai/event"><strong>3. Lablab AI</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxRZYm5lCKTSCGAWHvt2Ng.png" /><figcaption>Lablabai AI Hackathons</figcaption></figure><p>Lablab AI is a dynamic platform that empowers tech enthusiasts to engage in innovative AI projects. Their hackathons provide a unique space for both AI and tech professionals and individuals from diverse domains to collaborate and create groundbreaking AI solutions. With a strong focus on innovation and open collaboration, Lablab AI sets the stage for industry-changing AI technologies and fosters a fast-growing community of professionals and early adopters.</p><p><strong>4. </strong><a href="https://devpost.com/"><strong>Devpost</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jvvY9vb9dsypTe6hkAClFw.png" /><figcaption>Devpost Hackathons Page</figcaption></figure><p>Devpost, formerly ChallengePost, is a gateway to a world of software competitions and tech opportunities. It’s a hub for software engineers to participate in hackathons, showcase their skills, and find fulfilling work. Devpost powers hackathons both in-person and online, offering a diverse array of challenges with exciting prizes, while also serving as a platform for companies to connect with developers and provide job opportunities.</p><p>After completing the hackathons, make sure to showcase your experience on social platforms like LinkedIn. Share your hard work, the final product, and, if you’re lucky to be one of the winners, your earnings with your network. It’s a great way to let others know about your achievements and the effort you’ve put in.</p><p><a href="https://www.upwork.com/"><strong>5. Upwork</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z6QKBJEQLEvuA58sI1dc8w.png" /><figcaption>Top data freelancers on Upwork</figcaption></figure><p>By now, you should’ve completed several real-world projects from the previous platforms, now it’s time to tackle some real client projects. And the best to do this is to offer your skills as a freelancer.</p><p>Upwork is a freelancing platform that opens up countless opportunities for individuals looking to gain practical experience as data engineers, scientists, or analysts. Here, you can offer your skills and services to real-world clients, taking on projects that align with your areas of interest, which can help you expand your portfolio and gain valuable experience. As a student or self-taught enthusiast, this is an invaluable opportunity to apply your knowledge build a portfolio, and also make some money aside.</p><h4>🤖 Conclusion</h4><p>In a nutshell, as a data student, there are plenty of ways to apply what you’ve learned in class to real-world projects. Platforms like Forage and Omdena offer you hands-on experience and exposure in various data fields allowing you to grow your hard skills. Platforms like Lablab AI and Devpost offer you the chance to work in innovative hackathons with various people from different domains allowing you to grow your soft skills such as teamwork, communication, etc.</p><p>Once you’ve built up your skills, consider Upwork to take on real client projects. These experiences can enhance your portfolio, helping you land future opportunities in the data world. So, start creating, innovating, and learning. Your data future awaits! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=02260745837e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Escaping Tutorial Hell for Data Scientists]]></title>
            <link>https://medium.com/@hamza.lbelghiti/how-to-effectively-use-tutorials-to-create-unique-projects-fef211148d5e?source=rss-887d95bc7820------2</link>
            <guid isPermaLink="false">https://medium.com/p/fef211148d5e</guid>
            <category><![CDATA[youtube]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Hamza Belghiti]]></dc:creator>
            <pubDate>Mon, 29 May 2023 03:12:34 GMT</pubDate>
            <atom:updated>2023-11-24T18:49:15.081Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I Built a Unique Data Science &amp; Engineering Project Using 4 Different YouTube Tutorials</h4><p>In today’s fast-paced world, tutorials have become an invaluable resource for aspiring data scientists and engineers. They provide a structured approach to learning and offer step-by-step guidance on building projects. However, many people find themselves stuck in “tutorial hell”, endlessly consuming tutorials without making significant progress in creating their own projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*co0pWFXxZjk8xDnM0IbbOg.png" /><figcaption>Tutorial Hell</figcaption></figure><p>In this article, I will share my journey to successfully building a data science and data engineering project by effectively combining four YouTube tutorials. I successfully developed a project that analyzes data engineer job postings from Glassdoor and predicts salaries based on various factors.</p><h4>🌌 Introduction</h4><p>As I pursued my Master’s degree in Data Science &amp; Engineering and searched for a summer internship, I realized the importance of having strong data-related projects on my resume. Like many others, I found myself lost in a sea of tutorials, unsure of which technologies to learn and which projects to undertake. If you’re in a similar position, my first recommendation is to do a quick search on LinkedIn or any other online job search platform to understand the skills required for your desired job position. This will help you determine the project that aligns with those skills.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*ivj0nUm1rxSmyDfLdYpBFg.png" /><figcaption>Example of a job description of a Data Scientist from LinkedIn Jobs</figcaption></figure><p>Having identified the project requirements, like data science skills such as data collection using APIs or web scraping, cleaning data, analyzing and exploring data (EDA), building a model, and using machine learning algorithms for predictions, it also has to include some data engineering techniques like building ETLs (Extract, Load, Transform) pipelines and using data orchestration tools such as Airflow.</p><p>Now, let’s go over all the different tutorials and see what I have learned from them, what I found useful while building my project, and what I didn’t use and why.</p><h4>1️⃣ Tutorial N°1: The Inspiration</h4><p>The idea for my project first popped up when I watched this video from <a href="https://medium.com/u/58e0048b1d42">Luke Barousse</a> where he talked about how he uses Python as a data analyst. He made a Streamlit app that lists job requirements for aspiring data nerds so they can focus more on the in-demand skills required by recruiters.</p><p>I thought to myself, “Why not do the same but only with data engineering jobs in Morocco <em>(the country I am from! 🇲🇦)”</em>, as I was interested in a data engineering position in my summer internship.</p><p><strong>❌ Didn’t Use :</strong></p><ul><li><strong>Data Collection Method</strong>: Luke used SerpApi to get the details of the jobs, such as job titles, job descriptions, and salary estimates. I didn’t use this because of the high pricing of SerpApi <em>(50$ for 5,000 searches a month 🤔).</em></li><li><strong>Data Pipeline: </strong>Luke used Google Cloud Functions to automate his data pipeline, which is only useful if you are using an API, but in my case, I wanted to use a web scraper.</li></ul><p><strong>✅ Did Use :</strong></p><ul><li><strong>Data Cleaning &amp; EDA: </strong>The data cleaning part was useful, especially the part where he extracted the skill keywords from the job description and plotted them in the EDA part. In addition to skills keywords, I extracted required degrees (bachelor, master) and years of experience.</li><li><strong>Streamlit UI: </strong>The way he presented his findings in the Streamlit dashboard using the Altair library was very elegant and simple. I used a similar layout for my project.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FiNEwkaYmPqY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DiNEwkaYmPqY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FiNEwkaYmPqY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/927911d74f79a5ff731ce5fe51f2f3a0/href">https://medium.com/media/927911d74f79a5ff731ce5fe51f2f3a0/href</a></iframe><h4>2️⃣ Tutorial N°2: The Data Science Part</h4><p><strong>The</strong> second tutorial is from <a href="https://medium.com/u/6ee1f7466557">Ken Jee</a>, where he goes through building a data science project from ideation and planning to execution. The series goes over web scraping job listings from Glassdoor, data cleaning, EDA, model building, and deploying the model using Flask.</p><p><strong>❌ Didn’t Use :</strong></p><ul><li><strong>Model Production: </strong>At<strong> </strong>the end of the series, Ken put the salary prediction model into production using Flask. He made an API endpoint that takes in the job posting data and returns an expected salary. I didn’t use this method; as you’ll see later, there’s a much simpler way.</li></ul><p><strong>✅ Did Use:</strong></p><ul><li><strong>Web Scraping:</strong> This part was the trickiest because Glassdoor made some changes to their website and the Selenium scraper code didn’t work as well as in the video tutorial, so I had to do some research and change some bits to get it going. A second problem is that after page ~8, all the scraped job listings are duplicates of previous ones, so you are left with only ~200 unique job postings. You will see later a solution implemented using tutorial n°3.</li><li><strong>Data Cleaning, EDA, and Model Building:</strong> As I said in the beginning, these parts were very useful, very detailed, and clearly explained. I didn’t need to look anywhere else for those parts of my project.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGmW4F6MHqqs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGmW4F6MHqqs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FGmW4F6MHqqs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/565daab33c409705e5a7f42c99ff21c3/href">https://medium.com/media/565daab33c409705e5a7f42c99ff21c3/href</a></iframe><h4>3️⃣ Tutorial N°3: The Data Engineering Part</h4><p>This third project by <a href="https://medium.com/u/27ed83189c91">Darshil Parmar</a>, where he analyzes NYC Taxi data using GCP Storage, Compute Instance, Mage Data Pipeline Tool, BigQuery, and Looker Studio, was useful because it solved the problem of duplicated job listings from the second tutorial, remember! How? In this tutorial, Darshil uses Mage an alternative to airflow to build a data pipeline.</p><p><strong>❌ Didn’t use :</strong></p><ul><li><strong>GCP Services:</strong> In my case, I didn’t use the GCP instead I preferred to use AWS S3 to store my data as CSV files.</li></ul><p><strong>✅ Did use :</strong></p><ul><li><strong>Mage ETL: </strong>To bypass the duplicated job postings, and to increase the number of jobs, and have more information, I needed to filter the listings by date “Posted Last Week” and scrape the listings weekly so that jobs scraped two weeks before don’t appear again, we only get new job postings. This logic was implemented using Mage, see the project architecture below ⬇️.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWpQECq5Hx9g%3Fstart%3D3649%26feature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWpQECq5Hx9g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWpQECq5Hx9g%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/880ef7e642ae8df5729cef1461c85d76/href">https://medium.com/media/880ef7e642ae8df5729cef1461c85d76/href</a></iframe><h4>4️⃣ Tutorial N°4: Integration and Deployment</h4><p>In this last tutorial, @<strong>Patrick Loeber </strong>covers how to build a machine-learning web application from scratch in Python with Streamlit. He uses Stack Overflow survey data to build the ML model. The useful part for me is the part where he saves the final model using the Pickle library and integrates it into the Streamlit app.</p><p><strong>❌ Didn’t use :</strong></p><ul><li><strong>Data Collection &amp; Cleaning:</strong> At this stage, I already had my data cleaned and ready, and my model was trained and ready to go.</li></ul><p><strong>✅ Did use :</strong></p><ul><li><strong>Model Saving &amp; Prediction Page: </strong>The part where he saves the model and then uses it in the Streamlit app is interesting, and that’s the same way the “Predict Salary” page in my project works, with a few adjustments I added to handle edge cases and errors.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fxl0N7tHiwlw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dxl0N7tHiwlw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fxl0N7tHiwlw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed4b1d8513ab74cdda40204a24b586a4/href">https://medium.com/media/ed4b1d8513ab74cdda40204a24b586a4/href</a></iframe><h4>📝 Final Project Architecture</h4><p>Here’s the full project architecture, it includes different parts from the different tutorials and a unique touch from me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7qpb6n8b2QkzmWK7PZdvw.png" /></figure><h4>🛠️ Technologies Used</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*3shi2LTAeuZuk-liM-P0XQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pGaEkmOrqdR8oFfk10PBIw.png" /><figcaption>💻 Link to the Project: <a href="https://jobsglassdoor-dataengineers.streamlit.app/">https://jobsglassdoor-dataengineers.streamlit.app/</a></figcaption></figure><p>✏️ <strong>Quick Note:</strong> I mentioned earlier that I wanted to work on Data Engineer jobs in Morocco, but unfortunately, job posters don’t provide expected salaries, and even the descriptions are sometimes left blank. In addition to that, the number of jobs listed is so small (~20 per week), all of which compromised my initial idea, so I decided to go with the US job listings.</p><h4>☄️ Conclusion</h4><p>As the great Picasso once said: “Good artists copy, great artists steal.” It’s very normal to take inspiration from others, you don’t necessarily need to create something entirely original from scratch. Instead, take inspiration from existing ideas, refine them, and incorporate them into your own work and needs.</p><p>By effectively combining insights and techniques from multiple tutorials, I was able to overcome tutorial hell and create a unique data science and data engineering project. I hope this article serves as a roadmap for others seeking to escape tutorial hell and transform their learning into tangible projects. Remember, tutorials are powerful tools, but true growth comes from adapting and applying knowledge creatively. Embrace tutorials as a stepping stone to building your own unique projects and propelling your professional growth in the dynamic tech industry.</p><p>😼 <a href="https://github.com/Hamagistral/DataEngineers-Glassdoor">GitHub Repository</a> | 💻 <a href="https://jobsglassdoor-dataengineers.streamlit.app/">Demo App</a> | 📨 <a href="https://www.linkedin.com/in/hamza-elbelghiti/">Contact Me via LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fef211148d5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Information From YouTube Video Using Whisper and Langchain]]></title>
            <link>https://medium.com/@hamza.lbelghiti/how-openai-whisper-and-langchain-can-answer-any-question-you-have-from-a-youtube-video-278d04cc3460?source=rss-887d95bc7820------2</link>
            <guid isPermaLink="false">https://medium.com/p/278d04cc3460</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hamza Belghiti]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 15:43:56 GMT</pubDate>
            <atom:updated>2023-05-14T00:08:02.174Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build a Youtube Video Summarizer and a Question Answering App Using Whisper, Langchain, and Streamlit</h4><p>Have you ever found yourself going through a long YouTube video, trying to find the answer to a specific question? It can be a frustrating experience, especially when you’re short on time. This is where the power of AI comes in — we’re going to develop an app that allows you to quickly and easily find the answer you’re looking for, without watching the entire video.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fuuuv3ooY1WQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Duuuv3ooY1WQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fuuuv3ooY1WQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f015df69d1b7d8aeee74a4c11ba19ca6/href">https://medium.com/media/f015df69d1b7d8aeee74a4c11ba19ca6/href</a></iframe><p>With this app, you can simply ask the question you want to find the answer to, and within just a few minutes, you’ll have the information you need. Moreover, this tool is cost-effective, with a low cost of just $0.006 per minute of video <a href="https://platform.openai.com/account/usage"><em>(you can see the usage of your API Key here)</em></a>. In this article, you’ll learn how to use AI tools such as Whisper and Langchain as well as Streamlit for the UI of the application.</p><blockquote><strong>Update (14/05/2023) Version 2 of the GPTube :</strong></blockquote><blockquote>- Added a function that summarizes the video.</blockquote><blockquote>- Switched to gpt-3.5-turbo model for a lower cost.</blockquote><blockquote>- Changed the layout a bit to include a side bar describing the project, and the title and thumbnail of inputed video.</blockquote><h4>✏️ Let’s Design the App</h4><p>Let’s begin by understanding our problem first. We have a YouTube video that we don’t want or don’t have time to watch, but we still want to get information out of it in a short amount of time and at a low cost.</p><p>To do this, we first need a way to transcribe the YouTube video into text, and then we need to search in that text for the relevant parts to answer the question we have. Luckily, OpenAI and Langchain provided us with the necessary tools to execute this project.</p><figure><img alt="Architecture of an Question Answering App Using OpenAI API Whisper, Langchain, and Streamlit." src="https://cdn-images-1.medium.com/max/941/1*ymlIwq87IuauhAWb4CHDNw.png" /><figcaption>GPTube Application Architecture</figcaption></figure><p>🤖 <strong>What is </strong><a href="https://openai.com/research/whisper"><strong>Whisper</strong></a><strong>: </strong>Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. We show that the use of such a large and diverse dataset leads to improved robustness to accents, background noise, and technical language. Moreover, it enables transcription in multiple languages, as well as translation from those languages into English.</p><p>🔗 <strong>What is </strong><a href="https://python.langchain.com/en/latest/index.html"><strong>Langchain</strong></a><strong>: </strong>LangChain is a framework built around LLMs. We can use it for chatbots, <strong>G</strong>enerative <strong>Q</strong>uestion-<strong>A</strong>nswering (GQA), summarization, and much more. The core idea of the library is that we can <em>“chain”</em> together different components to create more advanced use cases around LLMs.</p><h4>🛠️ Let’s Build the App</h4><p>Now that we have an idea about the two main tools that we’re going to use and the structure of our app, let’s begin building.</p><p><strong>📌 Importing the libraries</strong></p><pre>import os <br>import re<br>import openai<br>import requests<br><br>from pytube.exceptions import VideoUnavailable<br>from urllib.parse import urlparse, parse_qs<br>from moviepy.editor import *<br>from pytube import YouTube<br><br>from langchain.chains.summarize import load_summarize_chain<br>from langchain.text_splitter import CharacterTextSplitter<br>from langchain.embeddings import OpenAIEmbeddings<br>from langchain.document_loaders import TextLoader<br>from langchain.docstore.document import Document<br>from langchain.vectorstores import Chroma<br>from langchain.chains import RetrievalQA<br>from langchain.llms import OpenAI</pre><p>First, we need to download the YouTube video into an mp3 file format using two libraries, <strong>pytube</strong> and <strong>moviepy</strong>. To keep our project directory clean, all the downloaded files will be saved in the temporary folder “tmp/”. The “tmp/” will only keep the transcribed txt files.</p><pre># Download YouTube video as Audio<br>def download_audio(url: str):<br><br>    yt = YouTube(url)<br><br>    # Extract the video_id from the url<br>    query = urlparse(url).query<br>    params = parse_qs(query)<br>    video_id = params[&quot;v&quot;][0]<br><br>    # Get the first available audio stream and download it<br>    audio_stream = yt.streams.filter(only_audio=True).first()<br>    audio_stream.download(output_path=&quot;tmp/&quot;)<br><br>    # Convert the downloaded audio file to mp3 format<br>    audio_path = os.path.join(&quot;tmp/&quot;, audio_stream.default_filename)<br>    audio_clip = AudioFileClip(audio_path)<br>    audio_clip.write_audiofile(os.path.join(&quot;tmp/&quot;, f&quot;{video_id}.mp3&quot;))<br><br>    # Delete the original audio stream<br>    os.remove(audio_path)</pre><p>As you can see, we defined our main function that’s going to generate the answer, We give as input the OpenAI API key <em>(</em><a href="https://platform.openai.com/account/api-keys"><em>get yours from here)</em></a>, the URL of the YouTube video, and the question we want the answer to.</p><p>Now that we have the mp3 file ready, we can do the transcription using the <strong>OpenAI Whisper API</strong>. However, there are some constraints to using Whisper such as the maximum file upload is currently limited to 25 MB and the transcription is only supported for some languages such as English, French, Arabic, and others <em>(</em><a href="https://platform.openai.com/docs/guides/speech-to-text/quickstart"><em>see the documentation for more.)</em></a></p><pre># Transcription <br>def transcribe_audio(file_path, video_id):<br>        # The path of the transcript<br>        transcript_filepath = f&quot;tmp/{video_id}.txt&quot;<br><br>        # Get the size of the file in bytes<br>        file_size = os.path.getsize(file_path)<br><br>        # Convert bytes to megabytes<br>        file_size_in_mb = file_size / (1024 * 1024)<br><br>        # Check if the file size is less than 25 MB<br>        if file_size_in_mb &lt; 25:<br>            with open(file_path, &quot;rb&quot;) as audio_file:<br>                transcript = openai.Audio.transcribe(&quot;whisper-1&quot;, audio_file)<br>                <br>                # Writing the content of transcript into a txt file<br>                with open(transcript_filepath, &#39;w&#39;) as transcript_file:<br>                    transcript_file.write(transcript[&#39;text&#39;])<br><br>            # Deleting the mp3 file<br>            os.remove(file_path)<br><br>        else:<br>            print(&quot;Please provide a smaller audio file (less than 25mb).&quot;)</pre><p>Now let’s use Langchain to search in the transcript for the relevant parts to answer the question. We’ll use a shorter version of the code to simplify things, but you’re welcome to see <a href="https://python.langchain.com/en/latest/modules/indexes/getting_started.html">the full code here</a>. We will be using the “Question answering over documents” use case as shown in the documentation, which combines a lot of different elements such as text splitters, embeddings, vector stores.</p><p>Question answering over documents consists of four steps:</p><ol><li>Create an index</li><li>Create a Retriever from that index</li><li>Create a question-answering chain</li><li>Ask questions!</li></ol><pre>   loader = TextLoader(transcript_filepath, encoding=&#39;utf8&#39;)<br>   documents = loader.load()<br>        <br>   texts = text_splitter.split_documents(documents)<br>   embeddings = OpenAIEmbeddings()<br>   db = Chroma.from_documents(texts, embeddings)<br><br>   retriever = db.as_retriever()<br>   qa = RetrievalQA.from_chain_type(llm=llm, chain_type=&quot;stuff&quot;, retriever=retriever)<br>   answer = qa.run(question)</pre><blockquote>Here’s how our function looks like after putting it all together :</blockquote><pre># Generate Answer<br>def generate_answer(api_key: str, url: str, question: str) -&gt; str:<br><br>    openai.api_key = api_key<br><br>    llm = OpenAI(temperature=0, openai_api_key=api_key, model_name=&quot;gpt-3.5-turbo&quot;)<br>    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)<br><br>    # Extract the video_id from the url<br>    query = urlparse(url).query<br>    params = parse_qs(query)<br>    video_id = params[&quot;v&quot;][0]<br><br>    # The path of the audio file<br>    audio_path = f&quot;tmp/{video_id}.mp3&quot;<br><br>    # The path of the transcript<br>    transcript_filepath = f&quot;tmp/{video_id}.txt&quot;<br><br>    # Check if the transcript file already exist<br>    if os.path.exists(transcript_filepath):<br>        <br>        loader = TextLoader(transcript_filepath, encoding=&#39;utf8&#39;)<br>        documents = loader.load()<br>        <br>        texts = text_splitter.split_documents(documents)<br>        embeddings = OpenAIEmbeddings()<br>        db = Chroma.from_documents(texts, embeddings)<br><br>        retriever = db.as_retriever()<br>        qa = RetrievalQA.from_chain_type(llm=llm, chain_type=&quot;stuff&quot;, retriever=retriever)<br>        answer = qa.run(question)<br><br>    else: <br>        download_audio(url)<br><br>        # Transcribe the mp3 audio to text<br>        transcribe_audio(audio_path, video_id)<br><br>        # Generating summary of the text file<br>        loader = TextLoader(transcript_filepath, encoding=&#39;utf8&#39;)<br>        documents = loader.load()<br>        <br>        texts = text_splitter.split_documents(documents)<br>        embeddings = OpenAIEmbeddings()<br>        db = Chroma.from_documents(texts, embeddings)<br><br>        retriever = db.as_retriever()<br>        qa = RetrievalQA.from_chain_type(llm=llm, chain_type=&quot;stuff&quot;, retriever=retriever)<br>        answer = qa.run(question)<br><br>    return answer.strip()</pre><blockquote><strong>Update (Summarizing the YouTube video) :</strong></blockquote><p>The same process of downloading and transcribing the video but using another chain specific to document summarizing.</p><pre># Generating Video Summary <br>def generate_summary(api_key: str, url: str) -&gt; str:<br><br>    openai.api_key = api_key<br><br>    llm = OpenAI(temperature=0, openai_api_key=api_key, model_name=&quot;gpt-3.5-turbo&quot;)<br>    text_splitter = CharacterTextSplitter()<br><br>    # Extract the video_id from the url<br>    query = urlparse(url).query<br>    params = parse_qs(query)<br>    video_id = params[&quot;v&quot;][0]<br><br>    # The path of the audio file<br>    audio_path = f&quot;tmp/{video_id}.mp3&quot;<br><br>    # The path of the transcript<br>    transcript_filepath = f&quot;tmp/{video_id}.txt&quot;<br><br>    # Check if the transcript file already exist<br>    if os.path.exists(transcript_filepath):<br>        # Generating summary of the text file<br>        with open(transcript_filepath) as f:<br>            transcript_file = f.read()<br><br>        texts = text_splitter.split_text(transcript_file)<br>        docs = [Document(page_content=t) for t in texts[:3]]<br>        chain = load_summarize_chain(llm, chain_type=&quot;map_reduce&quot;)<br>        summary = chain.run(docs)<br>    <br>    else: <br>        download_audio(url)<br><br>        # Transcribe the mp3 audio to text<br>        transcribe_audio(audio_path, video_id)<br><br>        # Generating summary of the text file<br>        with open(transcript_filepath) as f:<br>            transcript_file = f.read()<br><br>        texts = text_splitter.split_text(transcript_file)<br>        docs = [Document(page_content=t) for t in texts[:3]]<br>        chain = load_summarize_chain(llm, chain_type=&quot;map_reduce&quot;)<br>        summary = chain.run(docs)<br><br>    return summary.strip()</pre><p>Our base application is ready, now we’re going to use <a href="https://streamlit.io/">Streamlit</a> to quickly set up a web interface.</p><pre>import streamlit as st<br>import os<br><br>from gptube import generate_answer, generate_summary, video_info, is_valid_openai_key, is_valid_youtube_url, get_video_duration, calculate_api_cost<br><br>st.set_page_config(page_title=&quot;GPTube&quot;, page_icon=&#39;🎥&#39;)<br><br># App UI<br>def gptube_app():<br><br>    with st.sidebar:<br>        st.markdown(&quot;### 🎥 GPTube : Video Demo&quot;)<br><br>        st.video(&quot;https://youtu.be/vqThQVTgAp8&quot;)<br><br>        st.markdown(&quot;&quot;&quot;&lt;div style=&quot;text-align: justify;&quot;&gt;Have you ever found yourself going through a long YouTube video, trying to find the answer to a specific question? With GPTube, <br>                    you can simply ask the question you want to find the answer to, and in less than 2 minutes, <br>                    you can get the answer at a low cost of only $0.006 per minute of video. Or get a summary of the entire video<br>                    for just $0.009/minute.&lt;/div&gt;&quot;&quot;&quot;, unsafe_allow_html=True)<br>        <br>        st.markdown(&quot;#&quot;)<br><br>        st.markdown(&quot;😼 Source code on [GitHub](https://github.com/Hamagistral/GPTube)&quot;)<br>        st.markdown(&quot;🛠️ Made by [Hamza El Belghiti](https://www.linkedin.com/in/hamza-elbelghiti/)&quot;)<br><br><br>    st.header(&quot;🎥 GPTube : Your Shortcut to Video Insights&quot;)<br><br>    st.markdown(&#39;#&#39;) <br><br>    choice = st.radio(&quot;Please choose an option :&quot;, (&#39;Generate Summary&#39;, &#39;Generate Answer to a Question&#39;), horizontal=True)<br><br>    st.markdown(&#39;#&#39;) <br><br>    # OPENAI API KEY<br>    st.markdown(&#39;#### 🔑 Step 1 : Enter Your OpenAI API Key&#39;) <br>    openai_api_key = st.text_input(&quot;[Get Yours From the OPENAI Website](https://platform.openai.com/account/api-keys) : &quot;, placeholder=&quot;sk-***********************************&quot;, type=&quot;password&quot;)<br>    <br>    # Disable YouTube URL field until OpenAI API key is valid<br>    if openai_api_key:<br>        st.markdown(&#39;#### 📹 Step 2 : Enter the YouTube Video URL&#39;)<br>        youtube_url = st.text_input(&quot;URL :&quot;, placeholder=&quot;https://www.youtube.com/watch?v=************&quot;)<br>    else:<br>        st.markdown(&#39;#### 📹 Step 2 : Enter the YouTube Video URL&#39;)<br>        youtube_url = st.text_input(<br>            &quot;URL : &quot;,<br>            placeholder=&quot;Please enter a valid OpenAI API key first&quot;,<br>            disabled=True<br>        )<br><br>    if is_valid_youtube_url(https://medium.com/youtube_url):<br>        video_duration = get_video_duration(youtube_url)<br>        option = &#39;summary&#39; if choice == &#39;Generate Summary&#39; else &#39;answer&#39;<br>        api_call_cost = calculate_api_cost(video_duration, option)<br><br>        if video_duration &gt;= 4 and video_duration &lt;= 20:<br>            st.info(f&quot;The duration of the video is {video_duration} minutes. This will cost you approximately ${api_call_cost}&quot;)<br>            <br>            thumbnail_url, video_title = video_info(youtube_url)<br>            st.markdown(f&quot;#### {video_title}&quot;)<br>            st.image(f&quot;{thumbnail_url}&quot;, use_column_width=&#39;always&#39;)<br>            <br>        else:<br>            st.warning(&quot;Please enter a youtube video that is 4 minutes long at minimum and 20 minutes at maximum.&quot;)<br>    else:<br>        st.error(&quot;Please enter a valid YouTube video URL.&quot;)<br><br><br>    if choice == &quot;Generate Summary&quot;:<br>        if openai_api_key and youtube_url:<br>            if st.button(&quot;Generate Summary&quot;):<br>                if not is_valid_openai_key(openai_api_key):<br>                    st.markdown(f&quot;##### ❌ Please enter a valid OpenAI API key. &quot;)<br>                elif not youtube_url:<br>                    st.warning(&quot;Please enter the YouTube video URL.&quot;)<br>                else:<br>                    with st.spinner(&quot;Generating summary...&quot;):<br>                            # Call the function with the user inputs<br>                            summary = generate_summary(openai_api_key, youtube_url)<br><br>                    st.markdown(f&quot;#### 🤖 Summary :&quot;)<br>                    st.success(summary)<br>        else:<br>            st.warning(&quot;Please enter a valid OpenAI API and YouTube URL key first&quot;)<br><br>    elif choice == &quot;Generate Answer to a Question&quot;: <br>        if openai_api_key and youtube_url:<br>            st.markdown(&#39;#### 🤔 Step 3 : Enter your question&#39;)<br>            question = st.text_input(&quot;What are you looking for ?&quot;, placeholder=&quot;What does X mean ? How to do X ?&quot;)<br>        else:<br>            st.markdown(&#39;#### 🤔 Step 3 : Enter your question&#39;)<br>            question = st.text_input(&quot;What are you looking for ?&quot;, placeholder=&quot;Please enter a valid OpenAI API and YouTube URL key first&quot;, disabled=True)<br>            <br>        # Add a button to run the function<br>        if st.button(&quot;Generate Answer&quot;):<br>            if not is_valid_openai_key(openai_api_key):<br>                st.markdown(f&quot;##### ❌ Please enter a valid OpenAI API key. &quot;)<br><br>            elif not youtube_url:<br>                st.warning(&quot;Please enter the YouTube video URL.&quot;)<br>            elif not question:<br>                st.warning(&quot;Please enter your question.&quot;)<br>            else:<br>                os.environ[&quot;OPENAI_API_KEY&quot;] = openai_api_key<br><br>                with st.spinner(&quot;Generating answer...&quot;):<br>                    # Call the function with the user inputs<br>                    answer = generate_answer(openai_api_key, youtube_url, question)<br><br>                st.markdown(f&quot;#### 🤖 {question}&quot;)<br>                st.success(answer)<br>            <br>gptube_app()</pre><p>We’ve imported our core function <strong>generate_answer and generate_summary </strong>from the <strong>gptube.py</strong> file as well as some data-validating functions that check if the OpenAI API key is valid or if the YouTube video does exist, so we don&#39;t get errors when running our function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_G-AVJvefJQ8z4dBIy13_A.png" /><figcaption>Example Using Fireship latest video about Mojo Lang</figcaption></figure><figure><img alt="Demo of the app using Fireship’s latest video about AI as example" src="https://cdn-images-1.medium.com/max/1024/1*KfXDxgNbp-mY4MoNno9Lnw.png" /><figcaption>Old Version</figcaption></figure><h4>🛸 What’s Next?</h4><p>This project just touched the surface of what you can do with Whisper and Langchain, feel free to work around and tweak the code to create something unique. For example, this version of the code only accepts YouTube videos that are 20 minutes long at most. There are ways to bypass this constraint, but this will be the topic of a future blog post. If you have some web development skills, you could try to make out of this a Chrome extension to work directly on the YouTube page, for example, etc…</p><p>That’s it for today, I hope this helped you see the power of combining different AI tools. If you want to test this project for yourself, please see the full project on my Github and follow the README instructions.</p><p>😼 <a href="https://github.com/Hamagistral/GPTube">Full Project on GitHub</a> | 📹 <a href="https://www.youtube.com/watch?v=uuuv3ooY1WQ">YouTube</a> | 📨 <a href="https://www.linkedin.com/in/hamza-elbelghiti/">Contact Me via LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=278d04cc3460" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>