<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rahul dhanawade on Medium]]></title>
        <description><![CDATA[Stories by Rahul dhanawade on Medium]]></description>
        <link>https://medium.com/@theanalyticsguy?source=rss-2e269ad67a6a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CHv0m5qzgLi8nIvBRDXCGw.png</url>
            <title>Stories by Rahul dhanawade on Medium</title>
            <link>https://medium.com/@theanalyticsguy?source=rss-2e269ad67a6a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:40:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@theanalyticsguy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Everything you need to know about ETL as a Beginner]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://theanalyticsguy.medium.com/everything-you-need-to-know-about-etl-as-a-beginner-17cec8f5a6ee?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/2600/0*tNXXM6M1jkkwy1_u" width="5775"></a></p><p class="medium-feed-snippet">What is ETL?</p><p class="medium-feed-link"><a href="https://theanalyticsguy.medium.com/everything-you-need-to-know-about-etl-as-a-beginner-17cec8f5a6ee?source=rss-2e269ad67a6a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://theanalyticsguy.medium.com/everything-you-need-to-know-about-etl-as-a-beginner-17cec8f5a6ee?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/17cec8f5a6ee</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[etl-tool]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Sun, 25 Aug 2024 03:21:36 GMT</pubDate>
            <atom:updated>2024-08-25T03:21:36.110Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Applying Linear Regression to Predict Diabetes Progression]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://theanalyticsguy.medium.com/applying-linear-regression-to-predict-diabetes-progression-3fb1a0aa027d?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/600/1*UjRIuTADSdy6rB752xVvXg.png" width="600"></a></p><p class="medium-feed-snippet">Linear regression is a powerful machine learning algorithm used to model the linear relationship between a dependent variable and one or&#x2026;</p><p class="medium-feed-link"><a href="https://theanalyticsguy.medium.com/applying-linear-regression-to-predict-diabetes-progression-3fb1a0aa027d?source=rss-2e269ad67a6a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://theanalyticsguy.medium.com/applying-linear-regression-to-predict-diabetes-progression-3fb1a0aa027d?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fb1a0aa027d</guid>
            <category><![CDATA[prédiction]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[predictions]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Wed, 12 Jun 2024 16:08:22 GMT</pubDate>
            <atom:updated>2024-06-12T16:08:22.326Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Learning DSA for Data Engineering — Chapter 0]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://theanalyticsguy.medium.com/learning-dsa-for-data-engineering-chapter-0-f76a58c19146?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/2600/1*tFQsiaYCqb6dr6FC2TClDQ.jpeg" width="9579"></a></p><p class="medium-feed-snippet">Big O notation</p><p class="medium-feed-link"><a href="https://theanalyticsguy.medium.com/learning-dsa-for-data-engineering-chapter-0-f76a58c19146?source=rss-2e269ad67a6a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://theanalyticsguy.medium.com/learning-dsa-for-data-engineering-chapter-0-f76a58c19146?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/f76a58c19146</guid>
            <category><![CDATA[big-o]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-o-notation]]></category>
            <category><![CDATA[dsa-courses]]></category>
            <category><![CDATA[dsa-convention]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 12:22:27 GMT</pubDate>
            <atom:updated>2024-02-26T12:22:27.057Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building an ELT Pipeline with Python: Leveraging S3, Redshift, and DBT]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/art-of-data-engineering/building-an-elt-pipeline-with-python-leveraging-s3-redshift-and-dbt-e6a67392c841?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/2600/0*GW_l1LRmF6fu4Pvt" width="5568"></a></p><p class="medium-feed-snippet">Explore the Potential of DBT in Redshift Transformation</p><p class="medium-feed-link"><a href="https://medium.com/art-of-data-engineering/building-an-elt-pipeline-with-python-leveraging-s3-redshift-and-dbt-e6a67392c841?source=rss-2e269ad67a6a------2">Continue reading on Art of Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/art-of-data-engineering/building-an-elt-pipeline-with-python-leveraging-s3-redshift-and-dbt-e6a67392c841?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6a67392c841</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Fri, 16 Feb 2024 09:21:24 GMT</pubDate>
            <atom:updated>2024-02-17T07:50:48.066Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Docker for Data Engineers: Guide for Beginners and Data Engineers]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/towards-data-engineering/docker-for-data-engineers-guide-for-beginners-and-data-engineers-8425318dcd61?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/2600/0*f_tidsJd9VFb5EPQ" width="3280"></a></p><p class="medium-feed-snippet">You&#x2019;re a data engineer wrangling massive datasets, building complex pipelines, and trying to ensure everything runs smoothly&#x2026; on&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/towards-data-engineering/docker-for-data-engineers-guide-for-beginners-and-data-engineers-8425318dcd61?source=rss-2e269ad67a6a------2">Continue reading on Towards Data Engineering »</a></p></div>]]></description>
            <link>https://medium.com/towards-data-engineering/docker-for-data-engineers-guide-for-beginners-and-data-engineers-8425318dcd61?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8425318dcd61</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[docker-image]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[dataengineering-future]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 17:03:39 GMT</pubDate>
            <atom:updated>2024-02-06T07:07:58.939Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Why Understanding ETL is Crucial (and Why It’s Often Misunderstood)!]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://theanalyticsguy.medium.com/why-understanding-etl-is-crucial-and-why-its-often-misunderstood-bc0eec77ade0?source=rss-2e269ad67a6a------2"><img src="https://cdn-images-1.medium.com/max/2600/0*S6MnuFeJpY5essXa" width="7952"></a></p><p class="medium-feed-snippet">Photo by Timo Volz on Unsplash</p><p class="medium-feed-link"><a href="https://theanalyticsguy.medium.com/why-understanding-etl-is-crucial-and-why-its-often-misunderstood-bc0eec77ade0?source=rss-2e269ad67a6a------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://theanalyticsguy.medium.com/why-understanding-etl-is-crucial-and-why-its-often-misunderstood-bc0eec77ade0?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc0eec77ade0</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[etl-tool]]></category>
            <category><![CDATA[extract-data]]></category>
            <category><![CDATA[dataengineering-future]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Thu, 01 Feb 2024 01:12:32 GMT</pubDate>
            <atom:updated>2024-02-01T20:22:06.557Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Streamlined Data Processing: From API to S3 with AWS and Airflow]]></title>
            <link>https://medium.com/towards-data-engineering/streamlined-data-processing-from-api-to-s3-with-aws-and-airflow-5f00522d9591?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f00522d9591</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Tue, 30 Jan 2024 10:58:42 GMT</pubDate>
            <atom:updated>2024-02-02T05:21:03.903Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psh6SwYFVwejUD4YA2NQnA.jpeg" /></figure><p><a href="https://github.com/rahul1889/Streamlined-Data-Processing--From-API-to-S3-with-AWS-and-Airflow"><em>Github Code Link</em></a></p><p>Buckle up as we guide you through a hands-on, step-by-step process of building a slick data pipeline using AWS wonders, starring the ONS API as our data playground.</p><h3>Step 1: Grabbing Data from ONS API</h3><p>Let’s kick things off by diving into the ONS API. This powerhouse lets us snag datasets with just a bit of HTTP magic. Learn how to wield Python like a wizard, extracting JSON data and effortlessly transforming it into a data frame. Each JSON field becomes a sleek column, making our data crystal clear.</p><pre>import requests<br>import json<br>import pandas as pd<br>import s3fs<br><br>def extract_field(item, field_name, default=&#39;N/A&#39;):<br>    if item is not None and isinstance(item, dict):<br>        field_value = item.get(field_name, default)<br>        return field_value<br>    else:<br>        return default</pre><p>In this initial section, we set the stage for our data journey. We import essential libraries like requests for API interaction, json for handling JSON data, pandas for data manipulation, and s3fs for seamless interaction with AWS S3. The extract_field function becomes our handy tool for safely retrieving values from nested dictionaries</p><pre>URL = &quot;https://api.beta.ons.gov.uk/v1/datasets&quot;<br>response = requests.get(URL)<br>json_data = json.loads(response.text)<br><br>items = json_data[&quot;items&quot;]</pre><p>Here, we make a connection to the Office for National Statistics (ONS) API by sending a request to the specified URL. The API responds with a JSON payload containing valuable information. We extract the relevant data by loading the JSON content and isolating the ‘items’ key, laying the foundation for subsequent processing steps.</p><pre>data_list = []<br><br>for item in items:<br>    data = {<br>        &quot;ID&quot;: extract_field(item, &#39;id&#39;),<br>        &quot;Description&quot;: extract_field(item, &#39;description&#39;)<br>    }<br><br>    # Extract and append contacts<br>    for i, contact in enumerate(item.get(&#39;contacts&#39;, []), start=1):<br>        data[f&quot;Contact {i} Name&quot;] = extract_field(contact, &#39;name&#39;, &#39;Contact&#39;)<br>        data[f&quot;Contact {i} Email&quot;] = extract_field(contact, &#39;email&#39;, &#39;N/A&#39;)<br>        data[f&quot;Contact {i} Telephone&quot;] = extract_field(contact, &#39;telephone&#39;, &#39;N/A&#39;)<br><br>    # Extract and append keywords<br>    data[&quot;Keywords&quot;] = extract_field(item, &#39;keywords&#39;, &#39;No keywords available&#39;)<br><br>    # Extract and append links<br>    data[&quot;Editions Link&quot;] = extract_field(item[&#39;links&#39;][&#39;editions&#39;], &#39;href&#39;, &#39;No editions link available&#39;)<br>    data[&quot;Latest Version Link&quot;] = extract_field(item[&#39;links&#39;][&#39;latest_version&#39;], &#39;href&#39;, &#39;No latest version link available&#39;)<br>    data[&quot;Taxonomy Link&quot;] = extract_field(item[&#39;links&#39;][&#39;taxonomy&#39;], &#39;href&#39;, &#39;No taxonomy link available&#39;)<br><br>    # Extract and append methodologies link if present<br>    data[&quot;Methodologies Link&quot;] = extract_field(item[&#39;links&#39;].get(&#39;methodologies&#39;), &#39;href&#39;, &#39;No methodologies link available&#39;)<br><br>    # Extract and append methodology details<br>    for i, methodology in enumerate(item.get(&#39;methodologies&#39;, []), start=1):<br>        data[f&quot;Methodology {i} Title&quot;] = extract_field(methodology, &#39;title&#39;, &#39;N/A&#39;)<br>        data[f&quot;Methodology {i} National Statistic&quot;] = extract_field(methodology, &#39;national_statistic&#39;, &#39;N/A&#39;)<br>        data[f&quot;Methodology {i} Next Release&quot;] = extract_field(methodology, &#39;next_release&#39;, &#39;N/A&#39;)<br>        data[f&quot;Methodology {i} QMI&quot;] = extract_field(methodology, &#39;qmi&#39;, &#39;N/A&#39;)<br><br>    # Extract and append additional fields<br>    data[&quot;Release Frequency&quot;] = extract_field(item, &#39;release_frequency&#39;, &#39;No release frequency available&#39;)<br>    data[&quot;State&quot;] = extract_field(item, &#39;state&#39;, &#39;No state field available&#39;)<br>    data[&quot;Title&quot;] = extract_field(item, &#39;title&#39;, &#39;No title available&#39;)<br><br>    data_list.append(data)</pre><p>Then we iterate through the extracted data items, structuring the information into a more organized format. Each item is transformed into a dictionary, capturing essential fields such as ID, description, contacts, keywords, links, methodologies, and additional details. This structured data is stored in the data_list for further processing.</p><pre># Create DataFrame<br>df = pd.DataFrame(data_list)<br><br># Save to CSV<br>df.to_csv(&quot;s3:path&quot;)</pre><p>In this final section, we leverage the Pandas library to convert our structured data into a DataFrame. The DataFrame provides a tabular representation of the information, making it easy to work with and analyze. Subsequently, the data is stored in CSV format and uploaded to an S3 bucket.</p><h3>Step 2: Setting Up AWS EC2 and Airflow</h3><p>Time to set up camp in the cloud with AWS EC2. We’ll show you how to spin up an EC2 instance and install Airflow using some command-line kung fu. Once that’s done, upload your code to Airflow, where the real magic happens. Create Directed Acyclic Graphs (DAGs) for a smooth data dance.</p><p>Steps:</p><pre>Access AWS Console: Log in and go to EC2 service.<br><br>Launch Instance: Click &quot;Launch Instance.&quot;<br><br>Choose AMI: Select an Amazon Machine Image.<br><br>Select Instance Type: Choose instance specifications.<br><br>Review and Launch: Check configurations and click &quot;Launch.&quot;<br><br>Create Key Pair: If needed, create or select a key pair.<br><br>Launch Instance: Confirm and launch the instance.<br><br>Connect to Instance: Use SSH to connect to your instance.</pre><p>Open Command Prompt (Windows) or Terminal (Linux/Mac):</p><pre>cd path/to/your/key<br><br>ssh -i &quot;your-key.pem&quot; ubuntu@your-instance-ip</pre><p>This command establishes a secure SSH connection to your EC2 instance, allowing you to interact with it via the command line. Make sure to replace placeholder values with your actual key file name and EC2 instance IP</p><h3>Step 3: Installing Dependencies and Apache Airflow on EC2</h3><pre># Update package list<br>sudo apt-get update<br><br># Install Python3 pip<br>sudo apt install python3-pip<br><br># Install Apache Airflow using pip<br>sudo pip install apache-airflow<br><br># Install additional Python libraries<br>sudo pip install pandas<br>sudo pip install s3fs<br>sudo pip install requests<br>pip install jsonlib-python3</pre><p>Make sure to execute these commands in sequence on your EC2 instance to install the necessary dependencies and Apache Airflow</p><pre>#Start Airflow <br><br>Airflow standalone</pre><p>Ensure that you run this command in the Airflow home directory on your EC2 instance. you will see credentials to sign in into web UI</p><p>You can access the Airflow web UI at http://your-instance-ip:8080 in your web browser.</p><p>You will be prompted to log in. Use the following credentials:</p><pre>Username: admin<br>Password: Password</pre><p>You should now have access to the Airflow web UI and can explore and manage your DAGs (Directed Acyclic Graphs) and tasks. If you have any DAGs already defined, you’ll be able to see and trigger them from the UI.</p><h3>Step 4: Creating an Airflow DAG</h3><pre>from datetime import timedelta<br>from airflow import DAG<br>from airflow.operators.python_operator import PythonOperator<br>from datetime import datetime<br>from api_extraction import extract_field<br><br>default_args = {<br>    &#39;owner&#39;: &#39;airflow&#39;,<br>    &#39;depends_on_past&#39;: False,<br>    &#39;start_date&#39;: datetime(2021, 1, 1),<br>    &#39;email&#39;: [&#39;airflow@example.com&#39;],<br>    &#39;email_on_failure&#39;: False,<br>    &#39;email_on_retry&#39;: False,<br>    &#39;retries&#39;: 1,<br>    &#39;retry_delay&#39;: timedelta(minutes=1),<br>}<br><br>dag1 = DAG(<br>    &#39;ONS_API_Extraction&#39;, <br>    default_args=default_args,<br>    description=&quot;First Airflow project&quot;,<br>    schedule_interval=timedelta(days=1),  # Adjust the schedule interval as needed<br>)<br><br>task_1 = PythonOperator(<br>    task_id=&#39;data_extraction_from_api&#39;,<br>    python_callable=extract_field,<br>    dag=dag1,<br>)<br><br>task_1</pre><p>Make sure to adjust the start_date and schedule_interval parameters according to your requirements. Additionally, you may need to define the dependencies between tasks based on your workflow.</p><h3>Step 5: Storing Data in AWS S3</h3><p>Get ready to be blown away by Amazon S3, the data storage juggernaut. We’ll guide you through creating an S3 bucket and seamlessly integrating it into your data pipeline. Your code becomes the maestro, conducting a symphony that stores your DataFrame as a CSV in S3. Secure, scalable, and oh-so-cool.</p><p>I am Assuming you have created an S3 bucket, you need to provide the bucket path in your Python function code.</p><pre># Replace &#39;your-s3-bucket&#39; with the actual name of your S3 bucket<br>s3_bucket = &#39;your-s3-bucket&#39;<br><br># Example path in S3 where you want to store the CSV file<br>s3_path = f&#39;s3://{s3_bucket}/your/data/path/data.csv&#39;<br><br># Update your data storage code with the S3 path<br># For example, in your Pandas DataFrame to CSV conversion<br>df.to_csv(s3_path)</pre><p>Now, Moving DAG Script and Python Function Code to airflow in your preferred directory</p><pre># Step 1: Save the DAG Script and Python Function Code<br># Save your DAG script (e.g., my_ons_dag.py) and Python function code (e.g., api_extraction.py) in your preferred directory.<br><br># Step 2: Locate the Airflow DAGs Folder<br>AIRFLOW_HOME=&quot;/path/to/your/airflow&quot;<br>DAGS_FOLDER=&quot;${AIRFLOW_HOME}/dags&quot;<br>CODE_FOLDER=&quot;${AIRFLOW_HOME}/code&quot;<br><br># Step 3: Copy the DAG Script and Python Function Code (you also can open the file and copy paste code)<br>cp my_ons_dag.py &quot;${DAGS_FOLDER}&quot;<br>cp api_extraction.py &quot;${CODE_FOLDER}&quot;<br><br># Step 4: Verify in Airflow Web UI<br># Access the Airflow web UI to check if the new DAG appears.<br><br># Step 5: Trigger the DAG<br># Manually trigger the DAG in the Airflow web UI or wait for the scheduled run.</pre><h3>Step 5: Testing, Testing</h3><p>Before we wrap up this tech fiesta, let’s make sure everything’s rock-solid. Test your pipeline multiple times, validating each step to ensure a smooth ride. This final check ensures your creation is battle-ready and can handle anything the real world throws at it.</p><p>and Testing…</p><p>And there you have it! From ONS API to AWS S3, a simplified data pathway. No tech overwhelm, just a smooth, streamlined process. Ready to give it a try?</p><p><em>Making a Living out of writing, if it helped you. Just clap.</em></p><p><em>Happy Learning :)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f00522d9591" width="1" height="1" alt=""><hr><p><a href="https://medium.com/towards-data-engineering/streamlined-data-processing-from-api-to-s3-with-aws-and-airflow-5f00522d9591">Streamlined Data Processing: From API to S3 with AWS and Airflow</a> was originally published in <a href="https://medium.com/towards-data-engineering">Towards Data Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Web Scraping — Extracting Product Data from Amazon with Beautiful Soup in Python]]></title>
            <link>https://theanalyticsguy.medium.com/web-scraping-extracting-product-data-from-amazon-with-beautiful-soup-in-python-0f86163a9241?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f86163a9241</guid>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Sun, 31 Dec 2023 16:24:52 GMT</pubDate>
            <atom:updated>2024-02-01T20:22:28.476Z</atom:updated>
            <content:encoded><![CDATA[<p>Web Scraping — Extracting Product Data from Amazon with Beautiful Soup in Python</p><p>Our aim is to gather product titles, prices, and ratings from Amazon’s search results for men’s jackets. This data can be used for various purposes, such as:</p><ul><li>Price Comparison: Analyze price trends across different brands and styles.</li><li>Product Research: Identify popular products, features, and customer sentiments.</li><li>Market Analysis: Track changes in product offerings and pricing strategies.</li></ul><p>Tools of the Trade:</p><ul><li>Beautiful Soup: A Python library that parses HTML and XML documents, making it easy to navigate and extract data from web pages.</li><li>Pandas: A powerful data analysis library that excels in data manipulation and organization.</li><li>Requests: A library for making HTTP requests to websites, allowing us to fetch the HTML content we need to analyze.</li><li>NumPy: A library for numerical computing, providing support for array operations and data cleaning.</li></ul><p>Ready to Extract? Let’s Code!</p><p>Importing necessary Libraries</p><pre>from bs4 import BeautifulSoup<br>import requests<br>import pandas as pd<br>import numpy as np</pre><p>Functions for Data Extraction</p><pre>#function to extract product title<br><br>def get_title(soup):<br>    <br>    try:<br>        #outer tag object<br>        title = soup.find(&quot;span&quot;, attrs= {&quot;id&quot;: &#39;productTitle&#39;})<br>        <br>        #inner navigatable string object<br>        title_value = title.text<br>        <br>        #Title as string value<br>        title_string = title_value.strip()<br>        <br>    except AttributeError:<br>        title_string = &quot;&quot;<br>        <br>    return title_string<br><br><br>#Function to extract Product_Price<br><br>def Product_Price(soup):<br>    try:<br>        whole_price = soup.find(&quot;span&quot;, attrs={&quot;class&quot;: &#39;a-price-whole&#39;}).text<br>        <br>        fraction_price = soup.find(&quot;span&quot;, attrs={&quot;class&quot;: &#39;a-price-fraction&#39;}).text<br>        <br>        Product_Price = whole_price + fraction_price<br>        <br>    except:<br>        <br>        try:<br>            whole_price = soup.find(&quot;span&quot;, attrs={&quot;class&quot;: &#39;a-price-whole&#39;}).text<br>        <br>            Product_Price = whole_price<br>            <br>        except:<br>            Product_Price = &quot;&quot;<br>        <br>    return Product_Price<br><br><br>#Function to extract Product_rating<br><br>def Product_rating(soup):<br>    try:<br>        Product_rating = soup.find(&quot;span&quot;, attrs={&quot;class&quot;: &#39;a-size-base a-color-base&#39;}).text.strip()<br>        <br>    except AttributeError:<br>        Product_rating = &quot;&quot;<br>        <br>    return Product_rating</pre><ul><li>get_title(soup): Extracts the product title from the BeautifulSoup object.</li><li>Product_Price(soup): Extracts the product price (dealing with different price formats).</li><li>Product_rating(soup): Extracts the product rating.</li></ul><p>Main Script</p><pre>if __name__ == &#39;__main__&#39;:<br>    # Take all of below code part under main function</pre><p>Setting Up the Web Scraping Environment</p><pre>URL = &quot;https://www.amazon.co.uk/s?k=jackets+for+men&amp;crid=28L86GYH7MBJG&amp;sprefix=jackets+for+men%2Caps%2C129&amp;ref=nb_sb_noss_1&quot;<br>HEADERS = ({&#39;User_Agent&#39;: &#39;...&#39;, &#39;Accept_Language&#39;: &#39;en-US, en;q=0.5&#39;})<br>webpage = requests.get(URL, headers=HEADERS)<br>soup = BeautifulSoup(webpage.content, &quot;html.parser&quot;)</pre><ul><li>URL: The target Amazon page for men’s jackets.</li><li>HEADERS: HTTP request headers with user-agent information.</li><li>webpage: Fetching the HTML content of the webpage.</li><li>soup: Creating a BeautifulSoup object to parse the HTML content.</li></ul><p>Extracting Links</p><pre>Links = soup.find_all(&quot;a&quot;, attrs={&#39;class&#39;:&#39;a-link-normal s-no-outline&#39;})<br>Links_list = []<br>for link in Links:<br>    Links_list.append(link.get(&#39;href&#39;))</pre><ul><li>Using BeautifulSoup to find all &lt;a&gt; tags with the specified class.</li><li>Extracting the href attribute (link) from each tag.</li></ul><p>Data Dictionary Initialization</p><pre>d = {&quot;title&quot;:[], &quot;price&quot;:[], &quot;rating&quot;:[]}</pre><p>Initializing an empty dictionary to store extracted data.</p><p>Looping Through Links and Extracting Data</p><pre>for link in Links_list:<br>    new_webpage = requests.get(&quot;https://www.amazon.co.uk&quot; + link, headers=HEADERS)<br>    new_soup = BeautifulSoup(new_webpage.content, &quot;html.parser&quot;)<br><br>    d[&#39;title&#39;].append(get_title(new_soup))<br>    d[&#39;price&#39;].append(Product_Price(new_soup))<br>    d[&#39;rating&#39;].append(Product_rating(new_soup))</pre><ul><li>Looping through each product link.</li><li>Making a new HTTP request for each product to get detailed information.</li><li>Calling the functions to extract title, price, and rating and appending them to the dictionary.</li></ul><p>Creating a DataFrame and Cleaning Data</p><pre>amazon_df = pd.DataFrame.from_dict(d)<br>amazon_df[&#39;title&#39;].replace(&#39;&#39;, np.nan, inplace=True)<br>amazon_df = amazon_df.dropna(subset=[&#39;title&#39;])</pre><ul><li>Creating a Pandas DataFrame from the dictionary.</li><li>Replacing empty strings in the ‘title’ column with NaN.</li><li>Dropping rows with NaN values in the ‘title’ column.</li></ul><p>Saving the Data to CSV</p><pre>amazon_df.to_csv(&quot;amazon_data.csv&quot;, header=True, index=False)</pre><p>Saving the DataFrame to a CSV file named “amazon_data.csv” without including the index.</p><p>Code and output data available here —</p><p><a href="https://github.com/rahul1889/Web-Scraping---Extracting-Product-Data-from-Amazon-with-Beautiful-Soup-in-Python">GitHub - rahul1889/Web-Scraping---Extracting-Product-Data-from-Amazon-with-Beautiful-Soup-in-Python: I set out to discover the hottest and trendiest jackets for men available on Amazon. To accomplish this, I embarked on a web scraping adventure using Python and BeautifulSoup.</a></p><p>Thanks for reading:) Comment if any queries .</p><p>Making a Living out of writing, if it helped you. Just clap.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f86163a9241" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is LinkedIn Truly Benefiting You?]]></title>
            <link>https://theanalyticsguy.medium.com/is-linkedin-truly-benefiting-you-085a1b3699d5?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/085a1b3699d5</guid>
            <category><![CDATA[jobseeking]]></category>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[helping-others]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Mon, 18 Dec 2023 06:21:32 GMT</pubDate>
            <atom:updated>2023-12-18T06:21:32.277Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we conclude, let’s delve into the perspectives of two distinct groups in the professional realm.</p><p>On one side, we have the hiring posters. These individuals, genuinely seeking to assist others, often bring forth job opportunities. However, it’s crucial to acknowledge that many of them promote products or services that contribute to their financial gains.</p><p>Now, consider the course sellers. Often seen explaining introductory concepts and enticing learners to enroll in their courses, their primary focus is on making sales. While some are industry experts genuinely sharing their knowledge, many are profit-driven without ensuring the effectiveness of their courses.</p><p>What about daily knowledge sharing? Reflect on the last time you implemented an Excel trick or a code snippet from LinkedIn. Such content primarily serves those showcasing their work to reach their career goals, leaving its overall usefulness subjective.</p><p>Random pictures with unrelated content may catch your eye, but their relevance to professional strategies or code remains unclear.</p><p>Lastly, the unnecessary copying and pasting of knowledge from ChatGPT raises questions about authenticity and practical applicability. Without hands-on experience or real-world context, such content lacks substantial value.</p><p>On top of that, the doubt about accepting LinkedIn requests with personalized notes and the common difficulty in getting interviews make us wonder if the skills we showcase in our CVs are genuinely accurate. It raises the question: Are we honestly reflecting our abilities?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=085a1b3699d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why S3 as an intermediate storage for the transformed data in the AWS ETL.]]></title>
            <link>https://theanalyticsguy.medium.com/why-you-must-consider-using-s3-as-an-intermediate-storage-for-the-transformed-data-in-the-aws-etl-9efd69e1ebdd?source=rss-2e269ad67a6a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9efd69e1ebdd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <dc:creator><![CDATA[Rahul dhanawade]]></dc:creator>
            <pubDate>Sat, 16 Dec 2023 17:40:46 GMT</pubDate>
            <atom:updated>2024-02-01T20:22:58.248Z</atom:updated>
            <content:encoded><![CDATA[<p>There are various reasons why S3 is used as an intermediate store for the transformed data.</p><p><strong>Flexibility and Decoupling: </strong>The transformed data is stored in S3, which offers a decoupling between the loading and transformation phases. The freedom to handle and manage data at each pipeline level independently is made possible by this division.</p><p><strong>Data Recovery and Backup:</strong> S3 serves as a scalable and robust storage option. You have a backup copy of the data in case something goes wrong with the loading procedure by keeping the converted data on S3. You can restart the loading process without having to restart the ETL transformation if it fails.</p><p><strong>Performance Optimization:</strong> It is frequently more efficient to load data straight from S3 to Redshift(Target Database) than from an on-premises site or another cloud storage provider. AWS Redshift may use parallel processing to improve performance and include built-in features for effectively copying data from S3.</p><p><strong>Data Validation and Quality Checks:</strong> Before importing the converted data into Redshift, more validation and quality checks can be performed by storing it in S3. To make sure that the modified data satisfies the required requirements, you can run SQL queries, carry out data profiling, or employ other techniques.</p><p><strong>Keeping Records and Historical Data:</strong> S3 is a great option for archiving and long-term data storage. S3 offers a convenient and economical storage option if previous iterations of the converted data must be kept or if you wish to maintain a record of various iterations.</p><p><strong>Scalability: </strong>S3 is appropriate for situations where the volume of transformed data may be significant because it is built to manage large-scale data storage. The ETL pipeline can effectively manage expanding datasets because of its scalability.</p><p><em>An additional layer of abstraction and advantages are provided by using S3 in between, which helps to create a reliable and expandable ETL architecture.</em></p><p>Thanks for reading, see you soon :)</p><p>Making a Living out of writing, if it helped you. Just clap.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9efd69e1ebdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>