<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ankhipaul on Medium]]></title>
        <description><![CDATA[Stories by ankhipaul on Medium]]></description>
        <link>https://medium.com/@paulankhi89?source=rss-6b54c0f83e04------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wbDB6yKDhiAc64Ua0SRdJg.jpeg</url>
            <title>Stories by ankhipaul on Medium</title>
            <link>https://medium.com/@paulankhi89?source=rss-6b54c0f83e04------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:32:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@paulankhi89/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS S3 Multipart Upload/Download using Boto3 (Python SDK)]]></title>
            <link>https://medium.com/analytics-vidhya/aws-s3-multipart-upload-download-using-boto3-python-sdk-2dedb0945f11?source=rss-6b54c0f83e04------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dedb0945f11</guid>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[aws-sdk]]></category>
            <dc:creator><![CDATA[ankhipaul]]></dc:creator>
            <pubDate>Fri, 03 Jul 2020 17:16:43 GMT</pubDate>
            <atom:updated>2020-07-13T12:48:01.481Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*OPji094ZhoHv5X4AhI7DEA.png" /></figure><p>We all are working with huge data sets on a daily basis. Part of our job description is to transfer data with low latency :). Amazon Simple Storage Service (S3) can store files up to 5TB, yet with a single PUT operation, we can upload objects up to 5 GB only. Amazon suggests, for objects larger than 100 MB, customers should consider using the <em>Multipart Upload</em> capability.</p><p><em>AWS SDK, AWS CLI</em> and <em>AWS S3 REST API</em> can be used for Multipart Upload/Download. For CLI, read this <a href="https://aws.amazon.com/premiumsupport/knowledge-center/s3-multipart-upload-cli/">blog post</a>, which is truly well explained.</p><p>We will be using Python SDK for this guide. Before we start, you need to have your environment ready to work with <strong>Python</strong> and <strong>Boto3</strong>. If you haven’t set things up yet, <a href="https://medium.com/@paulankhi89/how-to-access-aws-s3-using-boto3-python-sdk-e5fbd3d276bd">please check out my previous blog post here</a>.</p><p>First, we need to make sure to import boto3; which is the Python SDK for AWS. Now create <strong>S3</strong> resource with <strong>boto3</strong> to interact with <strong>S3:</strong></p><pre>import boto3</pre><pre>s3_resource = boto3.resource(<strong>&#39;s3&#39;</strong>)</pre><p>When uploading, downloading, or copying a file or S3 object, the AWS SDK for Python automatically manages retries, multipart and non-multipart transfers. In order to achieve fine-grained control, the default settings can be configured to meet requirements. <strong>TransferConfig object</strong> is used to configure these settings. The object is then passed to a transfer method (upload_file, download_file) in the <em>Config= parameter</em>.</p><pre>from <strong>boto3.s3.transfer</strong> import <strong>TransferConfig</strong></pre><pre>config = <strong>TransferConfig</strong>(<strong>multipart_threshold</strong>=1024 * 25, <br>                        <strong>max_concurrency</strong>=10,<br>                        <strong>multipart_chunksize</strong>=1024 * 25,<br>                        <strong>use_threads</strong>=True)</pre><p>Here’s an explanation of each element of <strong>TransferConfig</strong>:</p><p><strong>multipart_threshold</strong>: This is used to ensure that multipart uploads/downloads only happen if the size of a transfer is larger than the threshold mentioned, I have used 25MB for example.</p><p><strong>max_concurrency</strong>: This denotes the maximum number of concurrent S3 API transfer operations that will be taking place (basically threads). Set this to increase or decrease bandwidth usage.This attribute’s default setting is 10.If <strong>use_threads</strong> is set to <strong>False</strong>, the value provided is ignored.</p><p><strong>multipart_chunksize</strong>: The size of each part for a multi-part transfer. Used 25MB for example.</p><p><strong>use_threads</strong>: If <strong>True</strong>, parallel threads will be used when performing <strong>S3</strong> transfers. If <strong>False</strong>, no threads will be used in performing transfers.</p><p>After configuring <strong>TransferConfig</strong>, lets call the S3 resource to upload a file:</p><pre>bucket_name = <strong>&#39;first-aws-bucket-1&#39;</strong></pre><pre>def multipart_upload_boto3():<br><br>    file_path = os.path.dirname(__file__) + <strong>&#39;/multipart_upload_example.pdf&#39;</strong></pre><pre><strong>    </strong>key = <strong>&#39;multipart-test/multipart_upload_example.pdf&#39;<br><br>    </strong>s3_resource.Object(bucket_name, key).upload_file(file_path,<br>                            ExtraArgs={<strong>&#39;ContentType&#39;</strong>: <strong>&#39;text/pdf&#39;</strong>},<br>                            Config=config,<br>                            Callback=ProgressPercentage(file_path)<br>                            )</pre><p>- <strong>file_path</strong>: location of the source file that we want to upload to s3 bucket.<br>- <strong>bucket_name</strong>: name of the destination S3 bucket to upload the file.<br>- <strong>key</strong>: name of the key (S3 location) where you want to upload the file.<br>- <strong>ExtraArgs</strong>: set extra arguments in this param in a json string. You can refer <a href="https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html">this</a> link for valid upload arguments.<br>- <strong>Config</strong>: this is the TransferConfig object which I just created above.</p><p>Similarly, for downloading file:</p><pre>def multipart_download_boto3():<br><br>    file_path = os.path.dirname(__file__)+ <strong>&#39;/multipart_download_example.pdf&#39;</strong></pre><pre><strong>    </strong>file_path1 = os.path.dirname(__file__)</pre><pre>    key = <strong>&#39;multipart-test/multipart_download_example.pdf&#39;<br><br>    </strong>s3_resource.Object(bucket_name, key).download_file(file_path,<br>                            Config=config,<br>                            Callback=ProgressPercentage(file_path1)<br>                            )</pre><p><strong>-bucket_name</strong>: name of the S3 bucket from where to download the file.<br>- <strong>key</strong>: name of the key (S3 location) from where you want to download the file(source).<br>-<strong>file_path</strong>: location where you want to download the file(destination)<br>-<strong>ExtraArgs</strong>: set extra arguments in this param in a json string. You can refer <a href="https://boto3.amazonaws.com/v1/documentation/api/1.9.42/guide/s3.html">this</a> link for valid upload arguments.<br>-<strong>Config</strong>: this is the TransferConfig object which I just created above.</p><p>Please note that I have used progress callback so that I can<br>track the transfer progress. Both the upload_file and<br>download_file methods take an optional callback parameter. This <em>ProgressPercentage</em> class is explained in <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/_modules/boto3/s3/transfer.html">Boto3 documentation</a>.</p><p><strong>Interesting facts of Multipart Upload (I learnt while practising):</strong></p><ol><li>In order to check the integrity of the file, before you upload, you can calculate the file’s MD5 checksum value as a reference. Say you want to upload a 12MB file and your part size is 5MB. Calculate 3 MD5 checksums corresponding to each part, i.e. the checksum of the first 5MB, the second 5MB, and the last 2MB. Then take the checksum of their concatenation. Since MD5 checksums are hex representations of binary data, just make sure you take the MD5 of the decoded binary concatenation, not of the ASCII or UTF-8 encoded concatenation. When that’s done, add a hyphen and the number of parts to get the <em>ETag </em>of the final object in S3.</li><li>For a traditional PUT request, <em>ETag</em> of the object is the MD5 checksum of the file. However, for multipart uploads <em>Etag</em> is calculated based on a different algorithm.</li></ol><p>Keep exploring and tuning the configuration of <strong>TransferConfig. </strong>Happy Learning!</p><p>For entire code reference, visit <a href="https://github.com/ankhipaul/aws_demos">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dedb0945f11" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/aws-s3-multipart-upload-download-using-boto3-python-sdk-2dedb0945f11">AWS S3 Multipart Upload/Download using Boto3 (Python SDK)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to access AWS S3 using Boto3 (Python SDK)]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-access-aws-s3-using-boto3-python-sdk-e5fbd3d276bd?source=rss-6b54c0f83e04------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5fbd3d276bd</guid>
            <category><![CDATA[aws-sdk]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python-boto-boto3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[boto3]]></category>
            <dc:creator><![CDATA[ankhipaul]]></dc:creator>
            <pubDate>Wed, 01 Jul 2020 22:20:41 GMT</pubDate>
            <atom:updated>2020-07-13T12:49:04.558Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*wECeMnlXFJTRAxXJIGiVoA.png" /></figure><p>In recent times, Amazon Web Services (AWS) has become quite popular in cloud computing. One of the main components is S3, the object storage service of AWS.</p><p>We can access S3 through AWS Console, AWS CLI and AWS SDKs of different languages. <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">Boto3</a> is the name of the Python SDK for AWS. If you’re working with S3 and Python, then you will know how cool the boto3 library is. It makes things <em>much</em> easier to work with.</p><h4><strong>Prerequisites</strong>:</h4><ol><li>Python 3+</li></ol><p>2. The boto3 module (<em>pip install boto3</em> to get it).</p><p>3. An AWS account with an AWS IAM user with <em>programmatic access</em>. Add <em>AmazonS3FullAccess</em> policy to that user. This is for simplicity, in prod you must follow the principal of least privileges.</p><p>4. Download the access key detail file from AWS console. Create a new file ~/.aws/credential on terminal and paste the details like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUepzPMKoZbSJ819NzV0xA.png" /></figure><p>5. Setup default region in ~/.aws/configfrom terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*Gcb5m460s65m4IfBb3YsGw.png" /></figure><h4><strong>Basic Operations</strong>:</h4><p>Now that we are ready, let’s start exploring some basic operations.<br><strong>Creating a Bucket</strong></p><p>The first important thing to remember is that the bucket name must be unique throughout the whole AWS platform, as bucket names are DNS compliant. If you try to use a name which another user has already claimed, your code will fail. Instead of success, you will see the following error: botocore.errorfactory.BucketAlreadyExists.</p><pre>import boto3<br>s3_resource = boto3.resource(<strong>&#39;s3&#39;</strong>)<br>s3_resource.create_bucket(Bucket=<strong>&quot;first-aws-bucket-1&quot;</strong>)</pre><p><strong>Listing all Buckets in S3</strong></p><p>Listing all the buckets present in S3 for that owner.</p><pre>for bucket in s3_resource.buckets.all():<br>    print(bucket.name)</pre><p>Output :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*XwZZYVcmIZ2xqus7P_Ky0Q.png" /></figure><p><strong>Uploading an Object</strong></p><p>Uploading a file<em> Screen_Shot.png</em> from local path<em> /</em>Users/ankhipaul/Documents/Screenshots to S3 bucket created above <em>first-aws-bucket-1</em></p><pre>s3_resource.Object(<strong>&#39;first-aws-bucket-1&#39;</strong>, <strong>&#39;Screen_Shot.png&#39;</strong>).\<br>    upload_file(Filename=<strong>&#39;/Users/ankhipaul/Documents/Screenshots/Screen_Shot.png&#39;</strong>)</pre><p>Output :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cm3pMvEq6wdGP7dBYW_Ulw.png" /></figure><p><strong>Downloading an Object from S3</strong></p><p>Downloading an object named <em>doc.pdf</em> from bucket <em>pythonusecase </em>to local system.</p><pre>s3_resource.Object(<strong>&#39;pythonusecase&#39;</strong>, <strong>&#39;doc.pdf&#39;</strong>).download_file(<br>    <strong>f&#39;/Users/ankhipaul/Documents/doc.pdf&#39;</strong>)</pre><p><strong>List all Objects of a specific Bucket</strong></p><p>Listing all Objects of the bucket <em>pythonusecase.</em> This will also list all the folders and the files of the respective folders inside this bucket.</p><pre>pythonusecase = s3_resource.Bucket(name = <strong>&#39;pythonusecase&#39;</strong>)<br>for object in pythonusecase.objects.all():<br>          print(object.key)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*vDlXrLbblJEZ0yrfqHPOwA.png" /></figure><p><strong>Copy an Object</strong></p><p>Copy the object <em>old_convertcsv.csv</em> to a new location within the same bucket with a new name <em>new_convertcsv.csv</em>. Thus, creating a new object.</p><pre>s3_resource.Object(<strong>&quot;pythonusecase&quot;</strong>, <strong>&quot;new_convertcsv.csv&quot;</strong>).copy_from(CopySource=<strong>&quot;pythonusecase/old_convertcsv.csv&quot;</strong>)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ccPHV3xKcz7qWHSw0gYPJQ.png" /></figure><p><strong>Deleting an Object</strong></p><p>Deleting the <em>old_convertcsv.csv </em>from bucket <em>pythonusecase</em></p><pre>s3_resource.Object(<strong>&quot;pythonusecase&quot;</strong>, <strong>&quot;old_convertcsv.csv&quot;</strong>).delete()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMkp2cB9R7QqRuTS__13rA.png" /></figure><blockquote><strong>Remember that S3 buckets do NOT have any “move” or “rename” operations. All we can do is create, copy and delete. As there is no move or rename; copy + delete can be used to achieve the same.</strong></blockquote><p><strong>Deleting a Bucket</strong></p><p>To remove buckets, we have to first make sure that buckets have no objects within them. First, delete all the objects of the bucket <em>first-aws-bucket-1. </em>Then delete the empty bucket.</p><pre>bucket = s3_resource.Bucket(<strong>&#39;first-aws-bucket-1&#39;</strong>)<br>bucket.objects.all().delete()<br>s3_resource.Bucket(<strong>&quot;first-aws-bucket-1&quot;</strong>).delete()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ho5DrYbzJYGULyhhEwn-cg.png" /><figcaption>Only one bucket is present in S3 after deletion</figcaption></figure><p><strong>Encryption of an Object</strong></p><p>S3 objects are encrypted using either server-side or client-side encryption. I will be using <em>Server-Side Encryption with Amazon S3-Managed Keys (SSE-S3)</em>; in this Amazon S3 encrypts an object using AES-256 algorithm before saving it to disk and decrypts it when you download the objects.</p><p>Uploading an image <em>random_pic.jpg</em> into bucket <em>pythonusecase</em> with AES-256 encryption.</p><pre>s3_resource.Object(<strong>&#39;pythonusecase&#39;</strong>, <strong>&#39;random_pic.jpg&#39;</strong>).\<br>    upload_file(Filename=<strong>&#39;/Users/ankhipaul/Documents/random_pic.jpg&#39;</strong>,ExtraArgs={<br>                         <strong>&#39;ServerSideEncryption&#39;</strong>: <strong>&#39;AES256&#39;</strong>})</pre><p>On AWS console, we can see the encryption as AES-256.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*dfAS6Ro0gx7-vte-X9b2uw.png" /></figure><p><strong>Versioning</strong></p><p>We have all used versioning to protect against accidental deletions or to maintain a complete record of our objects over time. When you request a versioned object, Boto3 will retrieve the latest version.</p><p>We have to first enable versioning. <em>In AWS, by default bucket versioning is disabled</em>. Once, we enable versioning, it can be suspended but cannot be disabled.</p><pre>s3_resource.BucketVersioning(<strong>&quot;pythonusecase&quot;</strong>).enable()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*Q5dNqsQJ9iOXqyBvZbBayA.png" /></figure><h4>Summary</h4><p>At this point, we’re done! You should now have a basic understanding of Boto3. For more details, visit <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html">https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html</a></p><p>For entire code reference, visit <a href="https://github.com/ankhipaul/aws_demos">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5fbd3d276bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-access-aws-s3-using-boto3-python-sdk-e5fbd3d276bd">How to access AWS S3 using Boto3 (Python SDK)</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>