<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amelia Tang on Medium]]></title>
        <description><![CDATA[Stories by Amelia Tang on Medium]]></description>
        <link>https://medium.com/@aimee.tang0317?source=rss-af5ef197b606------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xY_rMcZ582sgBNLuZRs_2Q.jpeg</url>
            <title>Stories by Amelia Tang on Medium</title>
            <link>https://medium.com/@aimee.tang0317?source=rss-af5ef197b606------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:45:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aimee.tang0317/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Create a Powerful ETL Data Pipeline with Python and AWS Services]]></title>
            <link>https://medium.com/@aimee.tang0317/how-to-create-a-powerful-etl-data-pipeline-with-python-and-aws-services-6ad8ddd7ca1b?source=rss-af5ef197b606------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ad8ddd7ca1b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[etl-pipeline]]></category>
            <dc:creator><![CDATA[Amelia Tang]]></dc:creator>
            <pubDate>Sat, 15 Jul 2023 21:32:43 GMT</pubDate>
            <atom:updated>2023-07-15T21:37:41.400Z</atom:updated>
            <content:encoded><![CDATA[<p>In data science, the ability to harness and extract insights from vast amounts of data is paramount. To accomplish this, data professionals rely on ETL (Extract, Transform, Load) data pipelines — a fundamental framework for streamlining data processing workflows. ETL pipelines facilitate the efficient extraction of data from diverse sources, transformation into a usable format, and loading into designated destinations for analysis.</p><p>For aspiring data scientists, mastering the art of building powerful data pipelines using Python and leveraging the capabilities of leading cloud computing platforms, such as Amazon Web Services (AWS), unlocks immense potential for scalable and efficient data processing.</p><p>In this blog post, I will demonstrate how I created an ETL data pipeline using Python, AWS EC2, S3, Glue and Athena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8HVnkz_Gt7Y_j0f9CNcfxQ.jpeg" /><figcaption>ETL Data Pipeline (Image from <a href="https://www.talend.com/resources/what-is-etl/">https://www.talend.com/resources/what-is-etl/</a>)</figcaption></figure><h3><strong>Overview</strong></h3><p><strong>Case: </strong>I designed the ETL data pipeline to enhance efficiency and accuracy in pricing used laptops for a second-hand computer and electronics store project. The management relies primarily on eBay completed or sold items as references for pricing their products. To build machine learning models for price prediction, I collected a large volume of used laptop raw data from eBay.</p><p><strong>Goal:</strong> Efficiently acquire and clean a large volume of used laptop data for further analysis</p><p><strong>ETL data pipeline diagram: </strong>The diagram provides a high-level overview of the flow and stages involved in data extraction, transformation, and loading, allowing stakeholders to understand the overall data processing workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*eMKSN1c9W3SyxlZLKo2b6Q.png" /><figcaption>ETL Data Pipeline Diagram (Image by Author)</figcaption></figure><h3><strong>Extract</strong></h3><p>The Python script I wrote to scrap public laptop listings from eBay.com and extract web data can be accessed <a href="https://github.com/aimee0317/used_laptop_price_predicter_deep_learning/blob/main/Medium/scraper.py">here</a>. It utilized the <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">BeautifulSoup library</a>, a popular Python library for web scraping that simplifies data extraction from HTML and XML documents.</p><p>I stored the Python script in an AWS S3 bucket and executed it on an AWS EC2 instance. Subsequently, I saved the output .csv file back to the same AWS S3 bucket. I have documented the entire process in another Medium post, which can be accessed <a href="https://medium.com/@aimee.tang0317/beginners-guide-to-aws-how-to-run-a-python-script-stored-in-s3-on-ec2-f05730c500e7">here</a>.</p><h3>Transform</h3><p><strong>Prerequisites:</strong></p><ul><li><strong>AWS account: </strong>a new account can be created <a href="https://aws.amazon.com/resources/create-account/">here</a>. The process is similar to signing up for any web services.</li></ul><p>Please note that while some AWS services offer free-tier options, AWS services, in general, are not free and usage of services may incur costs based on factors such as resource usage, data transfer, and storage.</p><ul><li><strong>AWS IAM: </strong>a web service that enables secure control and management of user access to AWS resources by providing authentication, authorization, and permission management capabilities.</li><li><strong>AWS Glue:</strong> a fully managed extract, transform, and load (ETL) service that makes it easy to prepare and load data for analytics by discovering, cataloging, and transforming data from various sources into a consistent format.</li></ul><p><strong>Step 1:</strong> <strong>Create an IAM role for AWS Glue</strong></p><p>First, navigate to AWS IAM and create a role that provides appropriate access permissions for AWS Glue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lMVj5pzEVBwo0VG-" /><figcaption>Create a new role in IAM (Image by Author)</figcaption></figure><p>For demonstration purposes and simplicity, I created a role with the PowerUserAccess policy, which provides users with full access to AWS services and resources, excluding IAM resource management. To grant more specific permissions to a role, other policies can be attached instead. For example, you can search for “Glue” in the search bar and choose the policy you wish to attach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*45xWLtKcQyNVDQsX" /><figcaption>Attach PowerUserAccess Policy (Image by Author)</figcaption></figure><p><strong>Step 2: Create an AWS Glue database</strong></p><p>After accessing AWS Glue, navigate to the “Data Catalog” section and click on “Databases”. Follow the web flow to add a new database, where typically you would only need to provide a name for the database and use the default settings for the remaining configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CzCpl2jMWdHtY3Au" /><figcaption>Create a database on AWS Glue (Image by Author)</figcaption></figure><p><strong>Step 3: Import .csv files from S3 to AWS Glue using a Crawler</strong></p><p>Once the database is created, click on the name of the database and proceed to create a Crawler. In AWS Glue, a crawler is an automated tool that scans and analyzes data sources to infer their schema and metadata, making it easier to catalog and process the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QtbffVkTDKObm_Oo" /><figcaption>Create an AWS Glue Crawler (Image by Author)</figcaption></figure><p>After clicking on “Create crawler,” select “Add a data source” and follow the web flow to complete the configuration of the crawler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3wHjyk0MFtLx5JRR" /><figcaption>Add a data source for Crawler (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*Z8px8bhwtLWK9_bP" /><figcaption>Choose S3 as the data source for Crawler (Image by Author)</figcaption></figure><p>Configure security settings for the Crawler and choose the IAM role we have created in <strong>Step 1.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EdGlgQQj_PRWAUs4" /></figure><p>Set the target database to the database we have created in <strong>Step 2.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H9a5JYhNkMJBm1oO" /><figcaption>Set target database for Crawler (Image by Author)</figcaption></figure><p>Follow the web flow to complete the creation of the Crawler. Once the Crawler is created, run it and wait for it to finish processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*32gFFneHywzoxAXU" /><figcaption>Run the Crawler (Image by Author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kDIEYJNLUST-EwHt" /><figcaption>Crawler Run Success (Image by Author)</figcaption></figure><p>Once the Crawler has completed running, you can view the table schema by clicking on “Tables” in the sidebar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vr3rZcWiHSs3H4md" /><figcaption>View Table Schema (Image by Author)</figcaption></figure><p><strong>Step 4: Transform the data using an AWS Glue job</strong></p><p>Next, we proceed to create an AWS Glue job to transform the data, making it ready for further analysis. Click on “ETL jobs” in the sidebar, and then select “Create job from a blank graph.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rmBdMw0uHMTNo0JY" /><figcaption>Create a Job From a Blank Graph in AWS Glue (Image by Author)</figcaption></figure><p>First, we need to create a data source, which is the .csv file the Crawler added to the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bb90ugCw5O8oEofT" /><figcaption>Add a Data Source for the Glue Job (Image by Author)</figcaption></figure><p>To transform the data to align with the specific needs and requirements of future analysis, AWS Glue jobs offer a variety of options. Some popular ones include:</p><ul><li><strong>ApplyMapping:</strong> Maps source columns to target columns.</li><li><strong>Filter: </strong>Filters data based on specified conditions to include or exclude certain records.</li><li><strong>Join:</strong> Combines data from multiple sources based on a common key or condition.</li></ul><p>For a comprehensive list of available options, you can refer to the official AWS documentation <a href="https://docs.aws.amazon.com/glue/latest/ug/edit-jobs-transforms.html">here</a>.</p><p>Here is a quick example showcasing the usage of <strong>ApplyMapping</strong> to change column names, modify data types, and drop specific columns during the data transformation process:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1p38UAHJ-B_V--yM" /><figcaption>Transform Data in Glue Job (Image by Author)</figcaption></figure><p>Next, we configure the target location for the transformed data as the S3 bucket, specifying the desired format such as .csv or .tsv.</p><p><strong>Please note that the S3 target location path must end with a trailing slash (/). To ensure this, click on “Browse S3,” select the desired bucket, and make sure to add a slash (/) after the automatically filled path.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e22pdaB5huHvYXyU" /><figcaption>Add Data Target in AWS Glue Job (Image by Author)</figcaption></figure><p>Then, we proceed to configure some basic job properties under “Job details.” We can utilize the same IAM role we created in <strong>Step 1</strong> since it already possesses the necessary access. For the remaining settings, I simply relied on the default options, but you can customize them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*eNWK7MnrgkrJt1nn" /><figcaption>Basic Properties under Job Details in AWS Glue (Image by Author)</figcaption></figure><p>Under “Job details,” we have the option to configure advanced properties, including saving the auto-generated Python script used for data transformation. This script can be stored in the same S3 bucket or any other preferred location of your choice.</p><p><strong>Please note that once again, it is essential to add a trailing slash (/) after selecting the path to avoid encountering an error message.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/0*jnMJ959s0NPkGZ45" /><figcaption>Save the Auto-generated Script to the S3 Bucket (Image by Author)</figcaption></figure><p>Lastly, continue following the remaining steps in the web flow to create the job. Once the job is successfully created, run it. To monitor the progress, you can click on “Job run monitoring” in the sidebar to view the status of the job execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*00gAq-L0JMILioxs" /><figcaption>Monitor AWS Glue Job Run Status (Image by Author)</figcaption></figure><p><strong>Step 5: Load the transformed data to Athena for query and analysis</strong></p><p>In the final step, we will load the transformed data into AWS Athena for querying and further analysis. To accomplish this, we need to create a new Crawler specifically for the transformed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1Z1Ums-Y0mtnfALh" /><figcaption>Create a New Crawler for the Transformed Data (Image by Author)</figcaption></figure><p>Create a new database for the transformed data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WaBi1XJj4-hhNzkb" /><figcaption>Create a Database for the Transformed Data (Image by Author)</figcaption></figure><p>Next, proceed with the web flow to create the database and follow the same process as <strong>Step 3</strong> to add the transformed data, which is saved in the S3 bucket, as a table in the database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4fcWr9ywMIQ1w_k1" /><figcaption>Load Transformed Data into AWS Athena (Image by Author)</figcaption></figure><p>Then, you can click on “Table data” to load the transformed data into AWS Athena. Additionally, we will need to configure the settings and add an IAM role for AWS Athena to ensure proper access and permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nBf0JLbZwyEqrdNq" /><figcaption>Configure Settings for AWS Athena (Image by Author)</figcaption></figure><p>Finally, we can now query and analyze the transformed data using SQL in AWS Athena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D2fmU4QxyCblOX7E" /></figure><p>Please note that AWS Athena is not entirely free, as it operates on a pay-per-query model. The costs are based on the amount of data scanned by your queries and the complexity of those queries. There may be opportunities to obtain some free usage based on your account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t9uz0dQsgSBeh5Rx794RmQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/ko/@eilisgarvey?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Eilis Garvey</a> on <a href="https://unsplash.com/photos/QoMQcDOadgk?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Congratulations on completing this demo! I hope that sharing my personal learning experience has provided valuable insights. Best wishes as you embark on your Data Science journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ad8ddd7ca1b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beginner’s Guide to AWS: How to Run a Python Script Stored in S3 on EC2]]></title>
            <link>https://medium.com/@aimee.tang0317/beginners-guide-to-aws-how-to-run-a-python-script-stored-in-s3-on-ec2-f05730c500e7?source=rss-af5ef197b606------2</link>
            <guid isPermaLink="false">https://medium.com/p/f05730c500e7</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Amelia Tang]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 18:44:05 GMT</pubDate>
            <atom:updated>2023-07-03T19:12:22.990Z</atom:updated>
            <content:encoded><![CDATA[<p>Running Python scripts on our local machines is often convenient and straightforward. However, there are instances where our machines may lack the necessary computing power or storage capacity. In such cases, leveraging cloud services becomes a viable solution.</p><p>Recently, I encountered a situation where I needed to run a web scraping Python script that would take hours to complete. To overcome this challenge, I opted to run the script on an AWS EC2 instance.</p><p>In this guide, I will provide a concise demonstration on how to upload a Python script to an S3 bucket, execute the script on an EC2 instance, and save the output file back to the S3 bucket for further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*YrgzB6aB4PPvE1skDO1DVw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/es/@carlheyerdahl?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Carl Heyerdahl</a> on <a href="https://unsplash.com/photos/KE0nC8-58MQ?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Step 1: Create an S3 Bucket</strong></p><p>First, after logging into the AWS Console, search for “IAM.”</p><p>IAM stands for Identity and Access Management. It helps manage the access to AWS resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IC0vlPJmDEbfFMT-6R80pw.png" /><figcaption>Search for IAM in AWS Console (Image by Author)</figcaption></figure><p>Then, click on “add users” to create a new user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLpPG6SywnU_-XaNW8MdQA.png" /><figcaption>Add Users in IAM (Image by Author)</figcaption></figure><p>To grant the user full access to S3, select “Attach policies directly” and search for “AmazonS3FullAccess”. Attach the policy to the user to provide full S3 access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQ9U2V0cF4e46CrPKH_zaA.png" /><figcaption>Set permissions in IAM (image by author)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d4ovsrvd5FVKhIRODvxiOQ.png" /><figcaption>Permissions Policies in IAM (Image by Author)</figcaption></figure><p>We can confirm that the new user has been successfully created. Now, let’s proceed to upload the Python script. To begin, click on the newly created user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*je5digsFDtV7Q_0FAV4Bvw.png" /><figcaption>IAM Users (Image by Author)</figcaption></figure><p>Under “Security Credentials” tab, we can find “Create access key.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_AmPLX9JS0S9P_Ot2cWKg.png" /><figcaption>Create Access Key in IAM (Image by Author)</figcaption></figure><p>Click on “Create access key” to download the access key and secret access key as a .csv file to your local machine. This file will serve as a reference for future use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bfHJuV4iWr7EbGRpxuFofg.png" /><figcaption>Access Keys in IAM (Image by Author)</figcaption></figure><p>Next, we import useful packages in our local Jupyter Notebook or Jupyter Lab and set the access key and the secret access key as constants.</p><pre>import logging<br>import boto3<br>from botocore.exceptions import ClientError<br><br>AWS_ACCESS_KEY_ID = access_key #It&#39;s in the file you saved in the previous step. Put it here in the string format. <br>AWS_SECRET_ACCESS_KEY = secret_access_key #It&#39;s in the file you saved in the previous step. Put it here in the string format.</pre><p>Next, we can create the S3 bucket using boto3. You can also create an S3 bucket without writing code. However, we will not delve into that process at this time.</p><pre>s3_client = boto3.client(&#39;s3&#39;, region_name = &#39;us-west-2&#39;, #choose the region you want. Here, &#39;us-west-2&#39; is an example.<br>                    aws_access_key_id = AWS_ACCESS_KEY_ID,<br>                    aws_secret_access_key = AWS_SECRET_ACCESS_KEY)<br><br>location = {&#39;LocationConstraint&#39;: &#39;us-west-2&#39;} <br><br>s3_client.create_bucket(Bucket = &#39;your-own-bucket-name&#39;, # Here is the name of the bucket you just created<br>                                  CreateBucketConfiguration = location)</pre><p><strong>Step 2: Upload the Python Script</strong></p><p>After creating the bucket, we can now upload the Python script:</p><pre>file_name = &#39;your-python-script.py&#39;<br>bucket_name = &#39;your-own-bucket-name&#39;<br>object_name = file_name #You can set your own object name or keep it the same as the original file name on the local machine.<br><br>with open(file_name, &#39;rb&#39;) as f:<br>    s3_client.upload_fileobj(f, bucket_name, object_name)</pre><p>When accessing the AWS console and navigating to the S3 bucket, we should see the uploaded script. Instead of the coding approach, as the root user or a user with appropriate access, you can simply click on the orange “Upload” button located in the upper right corner of the AWS console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kkX6SzNCzDaGaYNKcIn6A.png" /><figcaption>Upload Files to S3 (Image by Author)</figcaption></figure><p><strong>Step 3: Grant EC2 Instance Access to S3 Bucket</strong></p><p>First, we need to set up an EC2 instance with AWS Linux 2023 using the free-tier settings/configurations. The specific configuration of the EC2 instance can be tailored according to your requirements, but we won’t delve into the details of that process here.</p><p>To grant the EC2 instance access to all the files in the S3 bucket, we will navigate to IAM once again and create a role with full access to the S3 bucket. Alternatively, you can choose to grant access only to a specific S3 bucket. However, for the purpose of this demo, we will grant full access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPBj5rw4ejECNruG6pfAuQ.png" /><figcaption>Create Roles in IAM (Image by Author)</figcaption></figure><p>When creating the role, we select the appropriate configurations, considering that we will be assigning this role to an EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4iebOawJYNWsUIpgvb8dqw.png" /><figcaption>Trusted Entity Type in IAM (Image by Author)</figcaption></figure><p>And also grant full S3 access by adding the permission.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nAWGUS37zZoVcC7m0cpBew.png" /><figcaption>IAM Role Permissions (Image by Author)</figcaption></figure><p>Adding a name and description for the new role and create the role:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntOCrSnmQ9x-8UElPxDsBw.png" /><figcaption>Name and Description of IAM Role (Image by Author)</figcaption></figure><p>Then, we go back the the running EC2 instance, click on “Action” → “Security” → “Modify IAM role”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhrlLNxGt_v_etFLC8WebQ.png" /><figcaption>EC2 Modify IAM role (Image by Author)</figcaption></figure><p>Click on the arrow and select the IAM role we just created, and then click on “Update IAM role.” This ensures that the EC2 instance now has the necessary access to the S3 bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yGgzbH-fdsC-5uBa3UoYmg.png" /><figcaption>EC2 Update IAM Role (Image by Author)</figcaption></figure><p><strong>Step 4: Running the Python Script in EC2 using the AWS console</strong></p><p>First, we need to run the following commands after connecting to the running EC2 instance to perform updates and install the necessary packages required to run the Python script.</p><p>Please note that these commands provided are for demonstration purposes. Depending on the specific script you would like to run, you may need to use different commands.</p><p>To become the root user:</p><pre>sudo su </pre><p>Run updates:</p><pre>sudo yum update</pre><p>To see if it already has Python3 (Amazon Linux 2023 already has):</p><pre>Python3</pre><p>Install pip:</p><pre>yum install python3-pip</pre><p>Then we can install required packages using pip:</p><pre>python3 -m pip install package-name</pre><p>Then, we need to install AWS CLI.</p><p>You can use different commands for different operating systems you chose for your EC2 instance according to the <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">AWS documentation</a>. Mine is Linux.</p><pre>curl &quot;https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip&quot; -o &quot;awscliv2.zip&quot;<br>unzip awscliv2.zip<br>sudo ./aws/install</pre><p>To run the Python Script, we need to first copy the script from the S3 bucket using:</p><pre>aws s3 cp s3://your-own-bucket-name/your-python-script.py ./your-python-script.py</pre><p>Please note that in the second path, the dot (.) represents the current directory and should not be omitted.</p><p>Then, we can finally execute the Python script:</p><pre>python3 your-python-script.py</pre><p>If the Python script generates an output file that needs to be copied to the S3 bucket, you can use the following command as an example, assuming the file format is .csv:</p><pre>aws s3 cp ./output_file.csv s3://your-own-bucket-name/output_file.csv</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*KDKypu0XCUneyR6ZxezGLA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brettgarwood?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Brett Garwood</a> on <a href="https://unsplash.com/photos/uvvvKneSp_U?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Conclusions</strong></p><p>Congratulations! You’ve reached the end of this demo. I hope that sharing my personal learning experience with you has been helpful. Best of luck on your journey of learning Data Science! It’s truly a captivating field that will surely keep us engaged and fascinated.</p><p><strong>References:</strong></p><p>Install or update the latest version of the AWS CLI <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html</a></p><p>Mohammed Murtuza Qureshi, Working with AWS S3 Buckets using Python &amp; boto3[MOOC], Coursera, <a href="https://www.coursera.org/projects/working-with-aws-s3-buckets-using-python-boto3">https://www.coursera.org/projects/working-with-aws-s3-buckets-using-python-boto3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f05730c500e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>