<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SivaraamTK on Medium]]></title>
        <description><![CDATA[Stories by SivaraamTK on Medium]]></description>
        <link>https://medium.com/@sivaraamtk?source=rss-fb89ea19ee2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rS_Lbg2QoskH4hxNAIueAg.jpeg</url>
            <title>Stories by SivaraamTK on Medium</title>
            <link>https://medium.com/@sivaraamtk?source=rss-fb89ea19ee2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:08:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sivaraamtk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[SSH in Python using Paramiko]]></title>
            <link>https://medium.com/featurepreneur/ssh-in-python-using-paramiko-e08fd8a039f7?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e08fd8a039f7</guid>
            <category><![CDATA[scp]]></category>
            <category><![CDATA[paramiko]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ssh-tunnel]]></category>
            <category><![CDATA[featurepreneur]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Fri, 19 Aug 2022 15:17:26 GMT</pubDate>
            <atom:updated>2022-08-28T13:54:34.338Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with remote servers, there will be times when we will need to directly interact with them and when we will need to use SSH, the method typically used to access a remote machine and run commands, retrieve files or upload files, or transfer files from the remote machine to the local or vice versa using SFTP (Secure File Transfer Protocol) and SCP(Secure Copy Protocol).</p><p>But these can generally be used only through terminals and are difficult to automate. That’s where Paramiko comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mM4u4Qtjeu8BMZIVJu8kEw.png" /></figure><h4>Paramiko:</h4><p><a href="https://www.paramiko.org/">Paramiko</a> is a pure-Python implementation of the SSHv2 protocol, providing both client and server functionality. It provides the foundation for the high-level SSH library <a href="https://fabfile.org/">Fabric</a>, which is recommended to use for common client use-cases such as running remote shell commands or transferring files. Direct use of Paramiko itself is only intended for users who need advanced/low-level primitives or want to run an in-Python sshd.</p><p>The primary client of Paramiko as documented in the API is Paramiko.SSHClient. An instance of the Paramiko.SSHClient can be used to make connections to the remote server and transfer files.</p><p>As a client, you can authenticate yourself using a password or key and as a server, you can decide which users are allowed access and the channels you allow.</p><h4>Install Paramiko:</h4><p><strong>Run the following command in the terminal to install the paramiko library:</strong></p><pre>pip install paramiko</pre><h4><strong>Making a Connection:</strong></h4><pre>import paramiko</pre><pre>ssh_client =paramiko.SSHClient()</pre><pre>ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())</pre><pre>ssh_client.connect(hostname=’hostname’,port=’PORT’,username=’dummy’,password=’mypassword’)</pre><p>we use “<em>set_missing_host_key_policy()” </em>to let know Paramiko that you validate your trust with the machine for the first time you try to connect to the server. Parsing an instance of the <a href="http://docs.paramiko.org/en/2.1/api/client.html#paramiko.client.AutoAddPolicy">AutoAddPolicy</a> to set_missing_host_key_policy() changes it to allow any host.</p><p><strong>Alternatively for Password and Private key-based auth:</strong></p><pre>transport = paramiko.Transport()</pre><pre>transport.start_client()</pre><pre>transport.auth_password(USER_NAME, PASSWORD)</pre><pre>pkey = paramiko.RSAKey.from_private_key_file(PRIVATE_KEY_FILE) transport.auth_publickey(USER_NAME, pkey)</pre><p><strong>For Password and OTP-based auth:</strong></p><pre>transport = paramiko.Transport()</pre><pre>transport.start_client()</pre><pre>transport.auth_interactive(username, handler)</pre><pre>def handler(self, title, instructions, prompt_list):</pre><pre>  answers = []</pre><pre>  for prompt_, _ in prompt_list:</pre><pre>    prompt = prompt_.strip().lower()</pre><pre>    if prompt.startswith(‘password’):</pre><pre>      answers.append(PASSWORD)</pre><pre>    elif prompt.startswith(‘verification’):</pre><pre>      answers.append(OTP)</pre><pre>    else:</pre><pre>      raise ValueError(‘Unknown prompt: {}’.format(prompt_))</pre><pre>  return answers</pre><p>we use <a href="https://docs.paramiko.org/en/stable/api/transport.html#paramiko.transport.Transport.auth_interactive">the auth_interactive</a> method which accepts a handler for passing the values to the prompt. The custom handler that we write has the logic to pass the OTP or Password etc.. based on the output in the prompt. If we want to get the input to the handler from stdin instead then we can use the <a href="https://docs.paramiko.org/en/stable/api/transport.html#paramiko.transport.Transport.auth_interactive_dumb">auth_interactive_dumb</a> method.</p><blockquote><strong>**NOTE: </strong><em>Storing your credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use `.env` file and load it into the Python script but, even storing sensitive information in a plain text file is not very secure. The better and more secure way is to store these data in an encrypted store, like </em>Hashicorp Vault, Keywhiz, etc…</blockquote><h4>Running Commands Remotely:</h4><p>To run a command “<em>exec_command()”</em> is called on the SSHClient with the command passed. The response is returned as a tuple of (stdin, stdout, stderr)</p><p><strong>To list all the files in a directory:</strong></p><pre><em>stdin,stdout,stderr=ssh_client.exec_command(“ls”)</em></pre><pre><em>print(stdout.readlines())</em></pre><pre><em># → [u’Desktop\n’,u’Documents\n’,u’Downloads\n’,u’Videos\n’]</em></pre><pre><em>print(stderr.readlines())</em></pre><pre><em># → []</em></pre><p>type(stdin) and type(stdout) is ‘paramiko.channel.ChannelFile’</p><p>type(stderr) is class ‘paramiko.channel.ChannelStderrFile’</p><blockquote>According to <a href="http://docs.paramiko.org/">paramiko.org</a>, they are all python file-like objects.</blockquote><p>The stdin is a write-only file that can be used for commands requiring input.</p><p>The stdout file gives the output of the command.</p><p>The stderr gives the errors returned on executing the command. It will be empty if there is no error</p><p><strong>To list all the files in a directory with sudo:</strong></p><pre><em>stdin, stdout, stderr = ssh.exec_command(“sudo ls”)</em></pre><pre><em>stdin.write(‘mypassword\n’)</em></pre><pre><em>print (stdout.readlines())</em></pre><pre><em># → [u’Desktop\n’,u’Documents\n’,u’Downloads\n’,u’Videos\n’]</em></pre><h4>File Transfers:</h4><p>File transfers are handled by the paramiko.SFTPClient which you get from calling open_sftp() on an instance of Paramiko.SSHClient.</p><p><strong>Downloading a file from a remote machine:</strong></p><pre>ftp_client=ssh_client.open_sftp()</pre><pre>ftp_client.get(‘remotefileth’,’localfilepath’)</pre><pre>ftp_client.close()</pre><p><strong>Uploading file from local to a remote machine:</strong></p><pre>ftp_client=ssh.open_sftp()</pre><pre>ftp_client.put(‘localfilepath’,remotefilepath’)</pre><pre>ftp_client.close()</pre><h4>Conclusion:</h4><p>I came across Paramiko when I was looking into if we can directly SSH into an EC2 instance from Python using Boto3 and found it interesting and tested it out with DB-Tunneling in PostgreSQL. Give it a go and let me know what you think. You check out its repository <a href="https://github.com/paramiko/paramiko">here</a>.</p><p>!HAPPY-CODING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e08fd8a039f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/ssh-in-python-using-paramiko-e08fd8a039f7">SSH in Python using Paramiko</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Important Tools]]></title>
            <link>https://medium.com/featurepreneur/kubernetes-and-important-tools-for-it-4644d56a5b60?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4644d56a5b60</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[kind]]></category>
            <category><![CDATA[amazon-eks]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 15:59:31 GMT</pubDate>
            <atom:updated>2022-08-28T14:06:05.345Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*t6vNwzHMF1pbiGesWjky9g.png" /><figcaption>k8s</figcaption></figure><p>During the development of an Application, we will need to begin by gathering requirements, planning and modeling our application, implementing the model by coding it, and finally deploying it. While it can be debatable that modeling or coding is the hardest part, it is evident that deployment is the most crucial step, as it may make or break the entire project.</p><p>Nowadays, most projects are divided into different microservices and use Docker to containerize them, making them easy to manage and scale. But they are generally focused on one task and contextualized for that purpose. Organizing these microservices together for a whole application is very difficult and that’s where Kubernetes comes in.</p><h4>Kubernetes:</h4><p><a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">Kubernetes</a>, also known as K8s, is an open-source system for automating the deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience running production workloads at Google, combined with best-of-breed ideas and practices from the community. You can check its Repository <a href="https://github.com/kubernetes/kubernetes">here</a>.</p><p>Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available. We will look into some of the popular tools for Kubernetes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*MLkYg96iTKu9snYWTHZmnA.png" /></figure><h4>Kubectl:</h4><p>From a standard user’s viewpoint, <a href="https://kubernetes.io/docs/reference/kubectl/kubectl/">kubectl</a> is the control panel for Kubernetes. From a technical standpoint, Kubectl can be considered as a controller for the Kubernetes cluster manager. It helps you to run all Kubernetes operations. The Kubernetes kubectl CLI allows one to communicate with Kubernetes clusters. It can be mounted on any desktop or workstation, allowing us to remotely control our Kubernetes cluster. You can check its Repository <a href="https://github.com/kubernetes/kubectl">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*Kgvc7FOunydRbY0mMztymw.png" /></figure><h4>The Kubernetes Dashboard:</h4><p><a href="https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/">The Kubernetes Dashboard</a> is a web-based UI that allows administrators to perform basic operating tasks and review cluster events. You can use the dashboard as a quick tool for viewing and monitoring the cluster, submitting manifests, inspecting logs, and getting system information. The Kubernetes dashboard, which is maintained by the official Kubernetes team, is a general-purpose web application that you can deploy on your cluster. It consists of two main components: a front-end and a back-end. The dashboard provides a web view of your cluster and can be used for simple administrative tasks. You can check its Repository <a href="https://github.com/kubernetes/dashboard">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*Ha1jhADMjxFSqtPtkOXtMA.png" /></figure><h4>kind (Kubernetes IN Docker):</h4><p><a href="https://sigs.k8s.io/kind">kind</a> is a tool for running local Kubernetes clusters using Docker container “nodes”. It can create a Kubernetes cluster within minutes. kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI. With Kind as a tool to create a Kubernetes cluster, you can stop worrying about creating a Kubernetes cluster for local development. The cluster can be spun up with just a command. All the long process of bootstrapping the clusters ( like installing kubeadm, setting the swap size, joining the nodes, Installing cni, etc ) is handled by kind. Kind automatically bootstraps each node with kubeadm. You no longer have to maintain the control plane and worker node components. You can check its Repository <a href="https://github.com/kubernetes-sigs/kind">here</a>. If you are interested in trying them out, you can look into these articles by my friend @sudirsha02 <a href="https://medium.com/featurepreneur/kind-kubectl-installation-in-linux-216081b0c8d4">-&gt;</a> <a href="https://medium.com/featurepreneur/k8-microservices-app-with-kind-794da3f6f5c6">-&gt;</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*3CrXy1vbgHnJDs-VTZ-NBg.png" /></figure><h4>Minikube:</h4><p><a href="https://minikube.sigs.k8s.io/docs/start/">Minikube</a> is a tool that sets up a Kubernetes environment on a local device. It’s technically a Kubernetes distribution, but because it addresses a different type of use case than most other distributions (like Rancher, OpenShift, and EKS), it’s more common to hear folks refer to it as a tool rather than a distribution. Minikube <em>is</em> a version of Kubernetes, so it doesn’t really make sense to think of them as being binary opposites. Minikube is available on Windows, Linux, and OSX.</p><p>Easily launch a single-node Kubernetes cluster inside a virtual machine (VM) on your laptop. In just 5 minutes you can explore Kubernetes’ main features. Launch the Minikube dashboard straight from the box with just one command. Minikube works by setting up a Kubernetes cluster on a local machine. Minikube supports a flexible deployment mode based on the concept of “<a href="https://minikube.sigs.k8s.io/docs/drivers">drivers</a>.” Depending on the driver you choose for deployment, your node, or nodes, can run in one of several ways. The drivers are available for your installation depend on which system you are using to run Minikube. Which driver to choose depends on what level of performance and isolation you want from Minikube.</p><p>Minikube is not intended for hosting production-grade workloads. While it’s possible to set up multiple nodes with Minikube, Minikube’s focus is more on getting a single-node cluster set up easily and quickly than it is on helping you experiment with complex, multi-node clusters. Likewise, while Minikube supports basic ingress configurations, its options are limited. If you need complex ingress — or complex networking in general — Minikube is not the best choice. You can check its Repository <a href="https://github.com/kubernetes/minikube">here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*Eh2Pj-O_unQIswNx0P6gaw.png" /></figure><h4>Amazon EKS (Amazon Elastic Container Service for Kubernetes):</h4><p><a href="https://aws.amazon.com/eks/">Amazon EKS</a> is a managed service that allows you to run Kubernetes on AWS Cloud without having to set up, administer, or maintain your own control plane and nodes. EKS runs and scales the Kubernetes control plane to achieve high availability across different AWS Availability Zones. In Amazon EKS, control plane instances are automatically scaled based on load, and unhealthy control plane instances are detected and replaced, as well as automated version upgrades and patching will be done automatically. Amazon EKS can be integrated with other AWS services to provide various facilities. For example, worker nodes can be provisioned by AWS EC2 Instances, ECR (Elastic Container Registry) for Container Images, and VPC (Virtual Private Cloud) for isolating resources. You can use the <a href="https://docs.aws.amazon.com/eks/latest/userguide/eks-ug.pdf">Official User Guide</a> for reference.</p><h4>Conclusion:</h4><p>These are all the tools I have tested in Kubernetes and Hope you guys also find the same joy I found in learning and experimenting with Kubernetes. I guess that’s it for now. !HAPPY-CODING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4644d56a5b60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/kubernetes-and-important-tools-for-it-4644d56a5b60">Kubernetes Important Tools</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alacritty: The Fastest Terminal]]></title>
            <link>https://medium.com/featurepreneur/alacritty-the-fastest-terminal-e1fdc6c13d90?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1fdc6c13d90</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[alacritty]]></category>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Thu, 11 Aug 2022 14:58:47 GMT</pubDate>
            <atom:updated>2022-08-14T12:12:49.577Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction:</h4><p>Alacritty is a modern terminal emulator that comes with sensible defaults but allows for extensive configuration. It claims to be the fastest terminal emulator in existence. Integrating with other applications, rather than reimplementing their functionality, provides a flexible set of features with high performance. Using the GPU for rendering enables optimizations that aren’t possible without it. The supported platforms currently consist of BSD, Linux, macOS, and Windows.</p><p>The software is considered to be at a <strong>beta</strong> level of readiness; there are a few missing features and bugs to be fixed, but it is already used by many as a daily driver.</p><p>Here’s a link to Alacritty’s open source repository on <a href="https://github.com/alacritty/alacritty">GitHub</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*CktOsog1ZQSg_yq2vu2Vqw.png" /></figure><h4>About the project :</h4><p>Alacritty is the result of frustration with existing terminal emulators. Using vim inside tmux in many terminals was a particularly bad experience. None of them were ever quite fast enough. Even so, Linux does have some decent alternatives. For example, urxvt and st give good experiences. The major downside with those options is the difficulty of configuration and the inability to run on non-X11 platforms. The options for macOS are prolonged–especially with a full-screen terminal on a 4k monitor. None of these terminals are cross-platform–they are usually married to the windowing and font rendering APIs of their native platform.Alacritty aims to address these issues.</p><p>A set of values guides the project’s architecture and features:</p><ol><li><strong>Correctness:</strong> Alacritty should be able to properly render modern terminal applications like tmux and vim.</li><li><strong>Performance:</strong> Alacritty should be the fastest terminal emulator available anywhere.</li><li><strong>Appearance:</strong> Alacritty should have beautiful font rendering and look fantastic on all supported platforms.</li><li><strong>Simplicity:</strong> Alacritty should be conservative about which features it offers.</li><li><strong>Portability:</strong> Alacritty should support major operating systems including Linux, macOS, and Windows.</li></ol><h4>Features:</h4><p>This section will give an overview of Alacritty’s features beyond its terminal emulation capabilities.</p><ol><li><strong>Control Sequences -</strong>To get a list with supported control sequences take a look at <a href="https://github.com/alacritty/alacritty/blob/master/docs/escape_support.md">Alacritty’s escape sequence support</a>.</li><li><strong>Vi Mode</strong> <strong>— </strong>The vi mode allows moving around Alacritty’s viewport and scrolling back using the keyboard.</li><li><strong>Motion — </strong>The cursor motions are set up by default to mimic vi, however, they are fully configurable.</li><li><strong>Selection — </strong>One useful feature of vi modes is the ability to make selections and copy text to the clipboard.</li><li><strong>Search — </strong>Search allows you to find anything in Alacritty’s scrollback buffer.</li><li><strong>Vi Search -</strong>In vi mode the search is bound to / for forward and for backward search. This allows you to move around quickly and help with selecting content.</li><li><strong>Normal Search -</strong>During a normal search, you don’t have the opportunity to move around freely, but you can still jump between matches using Enter and Shift+Enter.</li><li><strong>Hints — </strong>Terminal hints allow easily interacting with visible text without having to start vi mode.</li><li><strong>Selection expansion -</strong>After making a selection, you can use the right mouse button to expand it.</li><li><strong>Opening URLs with the mouse -</strong>You can open URLs with your mouse by clicking on them.</li><li><strong>Multi-Window -</strong>Alacritty supports running multiple terminal emulators from the same Alacritty instance.</li><li><strong>OpenGL Rendering — </strong>Alacritty’s renderer is capable of doing ~500 FPS with a large screen full of text. This is made possible by efficient OpenGL usage. <a href="https://github.com/jwilm/alacritty">Alacritty</a> is very good at processing huge amounts of text.</li><li><strong>The Parser — </strong><a href="https://github.com/jwilm/alacritty">Alacritty</a>’s performance is enhanced by having a good parser, Rust helped significantly with this.</li><li><strong>Zero-cost abstractions — </strong>Rust’s zero-cost abstractions enable building nicely abstracted components and later combining them as if they were hand-written as one initially.</li><li><strong>New libraries — </strong>Developing <a href="https://github.com/jwilm/alacritty">Alacritty</a> required several pieces of library infrastructure which were not available. A non-GPL licensed cross-platform clipboard library, a vte parser, cross-platform font rasterization, and fontconfig bindings were all needed to build this project. <a href="https://docs.rs/vte">vte</a> and <a href="https://docs.rs/utf8parse">utf8parse</a> have been published on <a href="https://crates.io/">crates.io</a>. The remaining libraries are still in Alacritty’s source tree and will be published independently at some point.</li></ol><h3>Installation for Ubuntu:</h3><p>```</p><p>sudo add-apt-repository ppa:mmstick76/alacritty</p><p>sudo apt update</p><p>sudo apt install alacritty</p><p>sudo curl <a href="https://sh.rustup.rs/">https://sh.rustup.rs</a> -sSf | sh</p><p>```</p><p>or we can download it directly from <a href="https://snapcraft.io/install/alacritty/ubuntu">Snap</a></p><h4>Verdict:</h4><p>Alacritty is a great minimal terminal emulator that can replace most everyday terminals. But it’s far from being perfect, there are still more features to be added and bugs to be fixed. In my opinion, the multiple windows feature is still unusable right now and it’s to use alternatives like tmux. The speed difference between using Alacritty and normal terminal is not too noticeable. We can’t verify the usage of GPU. It’s no different from a standard terminal for those without a GPU. But I believe that this is just the beginning and recommend everyone with a GPU to try it. In the future, there will be some serious debates between users of Alacritty and Warp, a similar terminal available only on Mac for now. Till then I guess this is it. !HAPPY-CODING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1fdc6c13d90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/alacritty-the-fastest-terminal-e1fdc6c13d90">Alacritty: The Fastest Terminal</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using EC2 services using Boto3]]></title>
            <link>https://medium.com/featurepreneur/using-ec2-services-using-boto3-ad5453fe3bea?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad5453fe3bea</guid>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[boto3-python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[boto]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 19:31:52 GMT</pubDate>
            <atom:updated>2022-08-06T05:59:51.837Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction:</h4><p>The Amazon EC2 is a cloud service within Amazon Web Services(AWS) platform that allows building and managing virtual machines to support various application workloads. With Amazon EC2, you can create, resize or decommission instances at any time depending on the business requirements.</p><p>Boto3 is the Python SDK for AWS and can be used to directly interact with AWS resources from Python scripts. Refer to my previous articles to know more about <a href="https://medium.com/featurepreneur/boto3-aws-sdk-for-python-e7391b9901c5">Boto3</a> and <a href="https://medium.com/featurepreneur/control-aws-s3-using-boto3-df58e7038175">Boto3 for S3</a></p><p>In this article, we will look at how we can use the Boto3 library to perform various operations on AWS EC2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*NAMha7yPxb9n0awunRyUTg.png" /></figure><h4>Prerequisites:</h4><p>To use boto3 for EC2 we need the following:</p><ul><li>AWS Account Credentials (Access key, Secret key)</li><li>IAM User with full access to EC2</li><li>AWS CLI</li><li>python3</li><li>boto3</li></ul><p>To install AWS CLI, run the following command in your terminal:</p><pre>pip install awscli</pre><p>Similarly, to install Boto3, run the following command in your terminal:</p><pre>pip install boto3</pre><p>To configure the AWS environment, type the following command in your terminal:</p><pre>aws configure</pre><p>This command will prompt you to enter information to form a connection with your AWS account. For the Access key and Secret key, enter your AWS Access Key and AWS Secret Access Key of the IAM User with the required permissions. For the Default region name, enter the server region in which the instance you want to access is. If you haven’t created an instance, use “us-east-1”. For the Default output format enter “json”</p><p>Alternatively, you can also pass this information as parameters to the client()</p><pre>import boto3</pre><pre>client = boto3.client(</pre><pre>‘ec2’,</pre><pre>aws_access_key_id=”XXXXXXX”,</pre><pre>aws_secret_access_key=”YYYYYYY”,</pre><pre>region_name=”us-east-1&quot;</pre><pre>)</pre><blockquote><strong>**NOTE: </strong><em>Storing your AWS credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use `.env` file and load it into the Python script but even storing AWS Access and Secret Keys in a plain text file is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.</em></blockquote><h4>How to create an EC2 key pair?:</h4><p>A key pair consists of a private key and a public key. These are required to be able to securely access an EC2 instance. Amazon stores the public key and we store the private key.</p><pre>def create_key_pair():</pre><pre> ec2_client = boto3.client(“ec2”, region_name=”us-west-2&quot;)</pre><pre> key_pair = ec2_client.create_key_pair(KeyName=”ec2-key-pair”)</pre><pre> private_key = key_pair[“KeyMaterial”]</pre><pre># write private key to file with 400 permissions</pre><pre> with os.fdopen(os.open(“/tmp/aws_ec2_key.pem”, os.O_WRONLY |   os.O_CREAT, 0o400), “w+”) as handle:</pre><pre> handle.write(private_key)</pre><p>The function above creates an EC2 key pair with the name ec2-key-pair and then stores it in a file located at “<strong>/tmp/aws_ec2_key.pem”</strong> with the 400 permissions that will be needed when we use the private key to access the EC2 instance.</p><h4>Creating EC2 instance:</h4><p>To create one or more EC2 instances, you need to use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances">create_instances()</a> method of the EC2 resource. The simplest EC2 instance configuration might include the following arguments:</p><ul><li>MinCount — minimum number of EC2 instances to launch</li><li>MaxCount — maximum number of EC2 instances to launch</li><li>ImageId — the Amazon Machine Image which is used to launch your EC2 instance</li><li>InstanceType — Instance Type specifies how much CPU and RAM resources your EC2 instance should have</li><li>KeyName — SSH key name, which you’re going to use to get remote access to the EC2 instance</li><li>SecurityGroupIds: Security groups allow you to control access into and out of your EC2 instance</li><li>IamInstanceProfile: The name of the IAM profile that will be attached to the EC2 instance</li></ul><p>Such configuration will launch an EC2 instance in the default VPC:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>KEY_PAIR_NAME = &lt;your-keypair&gt;</pre><pre>AMI_ID = &lt;image-id&gt;</pre><pre>SUBNET_ID = &lt;subnet-id&gt;</pre><pre>SECURITY_GROUP_ID = &lt;security-group-id&gt;</pre><pre>INSTANCE_PROFILE = &lt;instance-profile-id&gt;</pre><pre>INSTANCE_TYPE = &lt;instance-type&gt;</pre><pre>USER_DATA = ‘’’#!/bin/bash</pre><pre>yum update</pre><pre>‘’’</pre><pre>EC2_RESOURCE = boto3.resource(‘ec2’, region_name=AWS_REGION)</pre><pre>EC2_CLIENT = boto3.client(‘ec2’, region_name=AWS_REGION)</pre><pre>instances = EC2_RESOURCE.create_instances(</pre><pre>MinCount = 1,</pre><pre>MaxCount = 1,</pre><pre>ImageId=AMI_ID,</pre><pre>InstanceType=INSTANCE_TYPE,</pre><pre>KeyName=KEY_PAIR_NAME,</pre><pre>SecurityGroupIds=[</pre><pre>SECURITY_GROUP_ID,</pre><pre>],</pre><pre>SubnetId=SUBNET_ID,</pre><pre>TagSpecifications=[</pre><pre>{</pre><pre>‘ResourceType’: ‘instance’,</pre><pre>‘Tags’: [</pre><pre>{</pre><pre>‘Key’: ‘Name’,</pre><pre>‘Value’: ‘my-ec2-instance’</pre><pre>},</pre><pre>]</pre><pre>},</pre><pre>]</pre><pre>)</pre><pre>for instance in instances:</pre><pre> print(f’EC2 instance “{instance.id}” has been launched)</pre><pre> instance.wait_until_running()</pre><pre> EC2_CLIENT.associate_iam_instance_profile(</pre><pre> IamInstanceProfile={</pre><pre> ‘Name’: INSTANCE_PROFILE,</pre><pre> },</pre><pre> InstanceId=instance.id,</pre><pre> )</pre><pre> print(f’EC2 Instance Profile “{INSTANCE_PROFILE}” has been attached’)</pre><pre>print(f’EC2 instance “{instance.id}” has been started’)</pre><p>Thecreate_instances() method returns a list of launched instances. You can use the for-loop to walk through the instances and wait till every instance is up and running if you need to (thewait_until_running() method).</p><h4>How to get the public IP for a running EC2 instance?:</h4><p>We can use the describe_instances method to retrieve the public IP of the EC2 instance:</p><pre>def get_public_ip(instance_id):</pre><pre> ec2_client = boto3.client(“ec2”, region_name=”us-west-2&quot;)</pre><pre> reservations = ec2_client.describe_instances(InstanceIds=[instance_id]).get(“Reservations”)</pre><pre> for reservation in reservations:</pre><pre>  for instance in reservation[‘Instances’]:</pre><pre>   print(instance.get(“PublicIpAddress”))</pre><p>After running this script for the instance id we had created above, I was able to retrieve the public IP of the instance. We can use this public IP and the key pair we created earlier to SSH into the instance.</p><h4>Listing EC2 Instances:</h4><p>The best way to list all EC2 instances is to use the all() method from the instances collection of the EC2 resource.</p><p>Then you can use for-loop to iterate through the returned list of instances to get the information about Instance ID (id), Platform (platform), Instance Type (instance_type), Public IP (public_ip_address), Image (image.id), and many others by accessing instance object attributes.</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>EC2_RESOURCE = boto3.resource(‘ec2’, region_name=AWS_REGION)</pre><pre>instances = EC2_RESOURCE.instances.all()</pre><pre>for instance in instances:</pre><pre> print(f’EC2 instance {instance.id}” information:’)</pre><pre> print(f’Instance state: {instance.state[“Name”]}’)</pre><pre> print(f’Instance AMI: {instance.image.id}’)</pre><pre> print(f’Instance platform: {instance.platform}’)</pre><pre> print(f’Instance type: “{instance.instance_type}’)</pre><pre> print(f’Public IPv4 address: {instance.public_ip_address}’)</pre><pre> print(‘-’*60)</pre><h4>Filtering EC2 instances:</h4><p>Filtering EC2 allows you to get a list of EC2 instances based on specified conditions. For example, you can get specific instance information by instance type, a list of instances based on Tags, instance state, and many other conditions.</p><p>For example for filtering EC2 instances by state, you can use the filter() method in the instances collection of the EC2 resource.</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>EC2_RESOURCE = boto3.resource(‘ec2’, region_name=AWS_REGION)</pre><pre>INSTANCE_STATE = ‘running’</pre><pre>instances = EC2_RESOURCE.instances.filter(</pre><pre>Filters=[</pre><pre>{</pre><pre>‘Name’: ‘instance-state-name’,</pre><pre>‘Values’: [</pre><pre>INSTANCE_STATE</pre><pre>]</pre><pre>}</pre><pre>]</pre><pre>)</pre><pre>print(f’Instances in state “{INSTANCE_STATE}”:’)</pre><pre>for instance in instances:</pre><pre> print(f’ — Instance ID: {instance.id}’)</pre><h4>Describing all EC2 instance properties:</h4><p>The describe_instances() method accepts the Filters and InstanceIds attributes that allow you to find specific instances.</p><pre>#!/usr/bin/env python3</pre><pre>import json</pre><pre>from datetime import date, datetime</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>EC2_CLIENT = boto3.client(‘ec2’, region_name=AWS_REGION)</pre><pre>INSTANCE_ID = &lt;instance-id&gt;</pre><pre># Helper method to serialize datetime fields</pre><pre>def json_datetime_serializer(obj):</pre><pre> if isinstance(obj, (datetime, date)):</pre><pre>  return obj.isoformat()</pre><pre>raise TypeError (“Type %s not serializable” % type(obj))</pre><pre>response = EC2_CLIENT.describe_instances(</pre><pre>InstanceIds=[</pre><pre>INSTANCE_ID,</pre><pre>],</pre><pre>)</pre><pre>print(f’Instance {INSTANCE_ID} attributes:’)</pre><pre>for reservation in response[‘Reservations’]:</pre><pre> print(json.dumps(</pre><pre>reservation,</pre><pre>indent=4,</pre><pre>default=json_datetime_serializer</pre><pre>)</pre><pre>)</pre><h4>Adding Tags to an EC2 instance:</h4><p>To add Tags to EC2 instances, you can use the create_tags() method of the EC2.Instance resource:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-2”</pre><pre>EC2_RESOURCE = boto3.resource(‘ec2’, region_name=AWS_REGION)</pre><pre>INSTANCE_ID = ‘i-020b3f1914105320d’</pre><pre>TAGS = [</pre><pre>{</pre><pre>‘Key’: ‘Environment’,</pre><pre>‘Value’: ‘dev’</pre><pre>}</pre><pre>]</pre><pre>instances = EC2_RESOURCE.instances.filter(</pre><pre>InstanceIds=[</pre><pre>INSTANCE_ID,</pre><pre>],</pre><pre>)</pre><pre>for instance in instances:</pre><pre> instance.create_tags(Tags=TAGS)</pre><pre> print(f’Tags successfully added to the instance {instance.id}’)</pre><h4>Getting EC2 instance state:</h4><p>To get the EC2 instance state, you can use the state list of the EC2.Instance resource:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = ‘us-east-2’</pre><pre>EC2_RESOURCE = boto3.resource(‘ec2’, region_name=AWS_REGION)</pre><pre>INSTANCE_ID = ‘i-020b3f1914105320d’</pre><pre>instance = EC2_RESOURCE.Instance(INSTANCE_ID)</pre><pre>print(f’EC2 instance “{INSTANCE_ID}” state: {instance.state[“Name”]}’)</pre><p>If you’d like to get a state of multiple EC2 instances, we can use the filter() method.</p><h4>How to run command in EC2 instance:</h4><pre>def execute_commands_on_linux_instances(client, commands, instance_ids):</pre><pre>&quot;&quot;&quot;<br>Runs commands on remote Linux instances</pre><pre>:param client: a boto/boto3 ssm client</pre><pre>: param commands: a list of strings, each one command to execute on the instances</pre><pre>: param instance_ids: a list of instance_id strings, of the instances on which to execute the command</pre><pre>:return: the response from the send_command function (check the boto3 docs for ssm client.send_command() )</pre><pre>&quot;&quot;&quot;</pre><pre>resp = client.send_command(</pre><pre>DocumentName=”AWS-RunShellScript”, # One of AWS’ preconfigured documents</pre><pre>Parameters={‘commands’: commands},</pre><pre>InstanceIds=instance_ids,</pre><pre>)</pre><pre>return resp</pre><pre># Example use:</pre><pre>ssm_client = boto3.client(‘ssm’) # Need your credentials here</pre><pre>commands = [‘echo “hello world”’]</pre><pre>instance_ids = [‘an_instance_id_string’]</pre><pre>execute_commands_on_linux_instances(ssm_client, commands, instance_ids)</pre><h4>How to stop an EC2 instance?:</h4><p>An EC2 instance can be shut down using the stop_instances command. A stopped EC2 instance is not charged for usage (except for any EBS volumes).</p><pre>def stop_instance(instance_id):</pre><pre> ec2_client = boto3.client(“ec2”, region_name=”us-west-2&quot;)</pre><pre> response = ec2_client.stop_instances(InstanceIds=[instance_id])</pre><pre> print(response)</pre><h4>How to terminate an EC2 instance?:</h4><p>We will use the terminate_instances method to terminate and remove our EC2 instance.</p><pre>def terminate_instance(instance_id):</pre><pre> ec2_client = boto3.client(“ec2”, region_name=”us-west-2&quot;)</pre><pre> response = ec2_client.terminate_instances(InstanceIds=[instance_id])</pre><pre> print(response)</pre><h4>Conclusion:</h4><p>In this article, we’ve provided many Python code snippets for creating, starting, stopping, rebooting, filtering, deleting, running commands in ec2, and tagging EC2 Instances using the Boto3. In addition to these, we can also perform management of SSH keys, Security Groups, Elastic IP addresses, and much more. If any doubt be sure to check out the documentation of <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html">Boto3 for ec2</a></p><p>! HAPPY_CODING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad5453fe3bea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/using-ec2-services-using-boto3-ad5453fe3bea">Using EC2 services using Boto3</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Useful methods for Boto3 in S3]]></title>
            <link>https://medium.com/featurepreneur/useful-methods-for-boto3-in-s3-611a15eab616?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/611a15eab616</guid>
            <category><![CDATA[boto3-python]]></category>
            <category><![CDATA[python-boto-boto3]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Sun, 31 Jul 2022 12:28:53 GMT</pubDate>
            <atom:updated>2022-08-06T06:00:17.594Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction:</h4><p>Boto3 is the name of the Python SDK for AWS. One of the core components of AWS is <a href="https://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a>, the object storage service offered by AWS. Using the Boto3 library with Amazon Simple Storage Service (S3) allows you to create, update, and delete S3 Buckets, Objects, S3 Bucket Policies, and many more from Python programs or scripts.</p><p>Here we will see how we can use the Boto3 module to perform various actions in S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*U3pDVWN-Xmr1MxUEmbzMkQ.png" /></figure><p>For more details on the Boto3 module check out my earlier articles <a href="https://medium.com/@sivaraamtk/control-aws-s3-using-boto3-df58e7038175">-&gt;</a> <a href="https://medium.com/@sivaraamtk/boto3-aws-sdk-for-python-e7391b9901c5">-&gt;</a>.</p><h4>Installation and Configuration:</h4><p>To use boto3 in S3 we need the following:</p><ul><li>AWS Account Credentials (Access key, Secret key)</li><li>IAM User with full access to S3</li><li>AWS CLI</li><li>python3</li><li>boto3</li></ul><p>To install AWS CLI, run the following command in your terminal:</p><pre>pip install awscli</pre><p>Similarly, to install Boto3, run the following command in your terminal:</p><pre>pip install boto3</pre><p>To configure the AWS environment, type the following command in your terminal:</p><pre>aws configure</pre><p>This command will prompt you to enter information to form a connection with your AWS account. For the Access key and Secret key, enter your AWS Access Key and AWS Secret Access Key of the IAM User with the required permissions. For the Default region name, enter the server region in which the bucket you want to access is. If you haven’t created a bucket or it is in a global region, use “us-east-1”. For the Default output format enter “json”</p><p>Alternatively, you can also pass this information as parameters to client()</p><pre>import boto3</pre><pre>client = boto3.client(</pre><pre>‘s3’,</pre><pre>aws_access_key_id=”XXXXXXX”,</pre><pre>aws_secret_access_key=”YYYYYYY”,</pre><pre>region_name=”us-east-1&quot;</pre><pre>)</pre><blockquote><strong>*NOTE: </strong><em>Storing your AWS credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use `.env` file and load it into the Python script but even storing AWS Access and Secret Keys in a plain text file is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.</em></blockquote><h4><strong>*** Important Fact *** :</strong></h4><p><em>S3 is object storage, it doesn’t have an actual directory structure. The “/” is rather cosmetic. So the names of files retrieved will include the full path too, and we can use that to identify the current location of that file.</em></p><h4>To list all files inside the current folder only :</h4><pre>import boto3</pre><pre>files_list = []</pre><pre>s3 = boto3.client(‘s3’)</pre><pre>object_list = s3.list_objects_v2(</pre><pre>Bucket=’bucket_name’,</pre><pre>Delimiter = ‘/’,</pre><pre>Prefix=’folder/sub-folder/’</pre><pre>)</pre><pre>#Do not use the prefix parameter if you are in the root directory</pre><pre>for item in object_list[‘Contents’]:</pre><pre>files = item[‘Key’]</pre><pre>file_name = files.replace(prefix,””)</pre><pre>#remove this check if you also want files in sub-folders</pre><pre>if ‘/’ not in file_name:</pre><pre> if ‘.’ in file_name and ‘/’ in files:</pre><pre>  files_list.append(files)</pre><pre>print files_list</pre><h3>To list all the only folders in the current directory:</h3><pre>import boto3</pre><pre>s3 = boto3.client(‘s3’)</pre><pre>response = s3.list_objects_v2(</pre><pre>Bucket = ‘bucket_name’,</pre><pre>Delimiter = ‘/’,</pre><pre>Prefix = ‘folder/sub-folder/’</pre><pre>)</pre><pre>#Do not use the prefix parameter if you are in the root directory</pre><pre>folder_list = []</pre><pre>for prefix in response[‘CommonPrefixes’]:</pre><pre> folder_name = prefix[‘Prefix’][:-1]</pre><pre> if folder_name != currentdir:</pre><pre>  folder_list.append(folder_name)</pre><pre>print folder_list</pre><h3>To Download a specific file:</h3><p><strong>**NOTE: </strong><em>Make sure the path to the folder or create the folder specified in output beforehand to avoid errors</em></p><pre>&quot;&quot;&quot;</pre><pre>output = location where to download the file</pre><pre>file = full file name with path prefix — the name of the file in the S3 bucket``</pre><pre>file_name = filename only — the name of the downloaded file to be saved</pre><pre>bucket = name of the S3 bucket</pre><pre>&quot;&quot;&quot;</pre><pre>s3 = boto3.resource(‘s3’)</pre><pre>file_name = str(file.rsplit(‘/’, 1)[-1])</pre><pre>output = f”downloads/{file_name}”</pre><pre>s3.Bucket(bucket).download_file(file, output)</pre><h3>To list all files of a specific type in the current folder and its sub-folders:</h3><pre>files_list = []</pre><pre>s3_resource = boto3.resource(‘s3’)</pre><pre>bucket = s3_resource.Bucket(‘bucket_name’)</pre><pre>if currentdir == ‘/’: #if current directory is root</pre><pre> objects = bucket.objects.all()</pre><pre>else:</pre><pre> objects = bucket.objects.filter(Prefix = currentdir)</pre><pre>ext_objects = [{‘Key’: o.key} for o in objects if o.key.endswith(file_extn)]</pre><pre>for item in ext_objects:</pre><pre> files_list.append(item[‘Key’])</pre><pre>print files_list</pre><h3>Conclusion:</h3><p>We can also upload and delete files similarly, and do much more. Most of the required methods are either already in the boto3 module (like implementing pagination, etc. ) or easy to define with the help of the existing methods. So be sure to give it a try and!CODE-HAPPY!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=611a15eab616" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/useful-methods-for-boto3-in-s3-611a15eab616">Useful methods for Boto3 in S3</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Control AWS S3 using Boto3]]></title>
            <link>https://medium.com/featurepreneur/control-aws-s3-using-boto3-df58e7038175?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/df58e7038175</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[boto3-python]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Sun, 24 Jul 2022 21:04:10 GMT</pubDate>
            <atom:updated>2022-08-04T14:01:51.644Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction :</h4><p>Boto3 is the name of the Python SDK for AWS. We can do the same things that we do in AWS Console and even more, but in a faster, repeated, and automated way. It allows you to directly create, update, and delete AWS resources from your Python scripts. I have written a detailed article about the <a href="https://medium.com/featurepreneur/boto3-aws-sdk-for-python-e7391b9901c5">Boto3 module</a> and <a href="https://medium.com/@sivaraamtk/useful-methods-for-boto3-in-s3-611a15eab616">more ways to use Boto3</a>, be sure to check it out before reading this.</p><p>One of the core components of AWS is <a href="https://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a>, the object storage service offered by AWS. With its impressive availability and durability, it has become the standard way to store videos, images, and data, commonly used for data analytics applications, machine learning, websites, and many more You can combine S3 with other services to build infinitely scalable applications.</p><p>Using the Boto3 library with Amazon Simple Storage Service (S3) allows you to create, update, and delete S3 Buckets, Objects, S3 Bucket Policies, and many more from Python programs or scripts with ease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*p-sLSVXiAoK78qREuBF5iA.png" /></figure><h4>Prerequisites :</h4><ul><li>AWS Account Credentials (Access key, Secret key)</li><li>IAM User with full access to S3</li><li>Install AWS CLI and configure</li><li>python3</li><li>Install boto3</li></ul><h4>Installation :</h4><p>To install AWS CLI, run the following command in your terminal:</p><pre>pip install awscli</pre><p>Similarly, to install Boto3, run the following command in your terminal:</p><pre>pip install boto3</pre><h4>Configuration :</h4><p>To configure the AWS environment, type the following command in your terminal:</p><pre>aws configure</pre><p>This command will prompt you to enter information to form a connection with your AWS account. For the Access key and Secret key, enter your AWS Access Key and AWS Secret Access Key of the IAM User with the required permissions. For the Default region name, enter the server region in which the bucket you want to access is. If you haven’t created a bucket or it is in global region, use “us-east-1”. For the Default output format enter “json”</p><p>Alternatively, you can also pass this information as parameters to the client()</p><pre>import boto3</pre><pre>client = boto3.client(</pre><pre>‘s3’,</pre><pre>aws_access_key_id=”XXXXXXX”,</pre><pre>aws_secret_access_key=”YYYYYYY”,</pre><pre>region_name=”us-east-1&quot;</pre><pre>)</pre><blockquote><strong>**NOTE: </strong><em>Storing your AWS credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use the `.env` file and load it into the Python script but even storing AWS Access and Secret Keys in a plain text file is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.</em></blockquote><h4>Create an S3 bucket using Boto3 :</h4><p>To create the Amazon S3 Bucket using the Boto3 library, you need to either <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.create_bucket">create_bucket client</a> or <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.ServiceResource.create_bucket">create_bucket resource</a>.</p><blockquote><strong>**NOTE</strong>: <em>Every Amazon S3 Bucket must have a unique name. Moreover, this name must be unique across all AWS accounts and customers.</em></blockquote><p>Creating S3 Bucket using Boto3 client:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>bucket_name = “demo-bucket”</pre><pre>location = {‘LocationConstraint’: AWS_REGION}</pre><pre>response = client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration=location)</pre><pre>print(“Amazon S3 bucket has been created”)</pre><blockquote><strong>**NOTE</strong>:<em> To avoid various exceptions while working with the Amazon S3 service, we strongly recommend you define a specific </em><a href="https://hands-on.cloud/aws-global-infrastructure-regions-and-availability-zones/"><em>AWS Region</em></a><em> for the Boto3 client and S3 Bucket Configuration</em></blockquote><p>Similarly, you can use the Boto3 resource to create an Amazon S3 bucket:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>bucket_name = “demo-bucket”</pre><pre>location = {‘LocationConstraint’: AWS_REGION}</pre><pre>bucket = resource.create_bucket(</pre><pre>Bucket=bucket_name,</pre><pre>CreateBucketConfiguration=location)</pre><pre>print(“Amazon S3 bucket has been created”)</pre><h4>Listing Amazon S3 Buckets using Boto3 :</h4><p>There are two ways of listing Amazon S3 Buckets:</p><ul><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_buckets">list_buckets()</a> — method of the client resource</li><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.ServiceResource.all">all()</a> — method of the S3 buckets resource</li></ul><p>Here’s an example of listing existing S3 Buckets using the S3 client:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>response = client.list_buckets()</pre><pre>print(“Listing Amazon S3 Buckets:”)</pre><pre>for bucket in response[‘Buckets’]:</pre><pre> print(f” — {bucket[‘Name’]}”)</pre><p>Here’s an example of listing existing S3 Buckets using the S3 resource:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>iterator = resource.buckets.all()</pre><pre>print(“Listing Amazon S3 Buckets:”)</pre><pre>for bucket in iterator:</pre><pre> print(f” — {bucket.name}”)</pre><h4>Deleting Amazon S3 Bucket using Boto3 :</h4><p>There are two possible ways of deletingAmazon S3 Bucket using the Boto3 library:</p><ul><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.delete_bucket">delete_bucket()</a> method of the S3 client</li><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Bucket.delete">delete()</a> method of the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#bucket">S3.Bucket</a> resource</li></ul><p>Here’s an example of deleting the Amazon S3 bucket using the Boto3 client:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>bucket_name = “demo-bucket”</pre><pre>client.delete_bucket(Bucket=bucket_name)</pre><pre>print(“Amazon S3 Bucket has been deleted”)</pre><p>Here’s an example of deleting the Amazon S3 bucket using the Boto3 resource:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>bucket_name = “demo-bucket”</pre><pre>s3_bucket = resource.Bucket(bucket_name)</pre><pre>s3_bucket.delete()</pre><pre>print(“Amazon S3 Bucket has been deleted”)</pre><h4>Deleting non-empty S3 Bucket using Boto3 :</h4><p>To delete an S3 Bucket using the Boto3 library, you have to clean up the S3 Bucket. Otherwise, the Boto3 library will raise the BucketNotEmpty exception. The cleanup operation requires deleting all S3 Bucket objects and their versions:</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_bucket = s3_resource.Bucket(S3_BUCKET_NAME)</pre><pre>def cleanup_s3_bucket():</pre><pre># Deleting objects</pre><pre> for s3_object in s3_bucket.objects.all():</pre><pre>  s3_object.delete()</pre><pre># Deleting objects versions if S3 versioning enabled</pre><pre> for s3_object_ver in s3_bucket.object_versions.all():</pre><pre>  s3_object_ver.delete()</pre><pre> print(“S3 Bucket cleaned up”)</pre><pre>cleanup_s3_bucket()</pre><pre>s3_bucket.delete()</pre><pre>print(“S3 Bucket deleted”)</pre><h4>Uploading a file to S3 Bucket using Boto3 :</h4><p>The Boto3 library has two ways for uploading files and objects into an S3 Bucket:</p><ul><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Bucket.upload_file">upload_file()</a> method allows you to upload a file from the file system</li><li><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.upload_fileobj">upload_fileobj()</a> method allows you to upload a file binary object data</li></ul><p>The upload_file() method requires the following arguments:</p><ul><li>file_name — filename on the local filesystem</li><li>bucket_name — the name of the S3 bucket</li><li>object_name — the name of the uploaded file (usually equals to the file_name)</li></ul><p>Here’s an example of uploading a file to an S3 Bucket:</p><pre>#!/usr/bin/env python3</pre><pre>import pathlib</pre><pre>import boto3</pre><pre>BASE_DIR = pathlib.Path(__file__).parent.resolve()</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>def upload_files(file_name, bucket, object_name=None, args=None):</pre><pre> if object_name is None:</pre><pre>  object_name = file_name</pre><pre>  s3_client.upload_file(file_name, bucket, object_name, ExtraArgs=args)</pre><pre>  print(f”’{file_name}’ has been uploaded to ‘{S3_BUCKET_NAME}’”)</pre><pre>upload_files(f”{BASE_DIR}/files/demo.txt”, S3_BUCKET_NAME)</pre><p>We’re using the pathlib module to get the script location path and save it to the BASE_DIR variable. Then, we’re creating the upload_files() method that is responsible for calling the S3 client and uploading the file.</p><h4>Uploading multiple files to the S3 bucket :</h4><p>To upload multiple files to the Amazon S3 bucket, you can use the glob() method from the glob module. This method returns all file paths that match a given pattern as a Python list. You can use glob to select certain files by a search pattern by using a wildcard character:</p><pre>#!/usr/bin/env python3</pre><pre>import os</pre><pre>import pathlib</pre><pre>from glob import glob</pre><pre>import boto3</pre><pre>BASE_DIR = pathlib.Path(__file__).parent.resolve()</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>S3_CLIENT = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>def upload_file(file_name, bucket, object_name=None, args=None):</pre><pre> if object_name is None:</pre><pre>  object_name = file_name</pre><pre>  S3_CLIENT.upload_file(file_name, bucket, object_name, ExtraArgs=args)</pre><pre>  print(f”’{file_name}’ has been uploaded to ‘{S3_BUCKET_NAME}’”)</pre><pre>files = glob(f”{BASE_DIR}/files/*.txt”)</pre><pre>for file in files:</pre><pre> upload_file(file, S3_BUCKET_NAME)</pre><h4>Uploading generated file object data to S3 Bucket using Boto3 :</h4><p>If you need to upload file object data to the Amazon S3 Bucket, you can use the upload_fileobj() method. This method might be useful when you need to generate file content in memory (example) and then upload it to S3 without saving it on the file system.</p><blockquote><strong>**NOTE</strong>: <em>the upload_fileobj() method requires opening a file in binary mode.</em></blockquote><p>Here’s an example of uploading a generated file to the S3 Bucket:</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>def upload_generated_file_object(bucket, object_name):</pre><pre> with io.BytesIO() as f:</pre><pre>  f.write(b’First line.\n’)</pre><pre>  f.write(b’Second line.\n’)</pre><pre>  f.seek(0)</pre><pre>s3_client.upload_fileobj(f, bucket, object_name)</pre><pre>print(f”Generated has been uploaded to ‘{bucket}’”)</pre><pre>upload_generated_file_object(S3_BUCKET_NAME, ‘generated_file.txt’)</pre><h4>Enabling S3 Server-Side Encryption (SSE-S3) for uploaded objects :</h4><p>You can use S3 Server-Side Encryption (SSE-S3) encryption to protect your data in Amazon S3. We will use server-side encryption, which uses the AES-256 algorithm:</p><pre>#!/usr/bin/env python3</pre><pre>import pathlib</pre><pre>import boto3</pre><pre>BASE_DIR = pathlib.Path(__file__).parent.resolve()</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>def upload_files(file_name, bucket, object_name=None, args=None):</pre><pre> if object_name is None:</pre><pre>  object_name = file_name</pre><pre> s3_client.upload_file(file_name, bucket, object_name, ExtraArgs=args)</pre><pre> print(f”’{file_name}’ has been uploaded to ‘{S3_BUCKET_NAME}’”)</pre><pre>upload_files(</pre><pre>f”{BASE_DIR}/files/demo.txt”,</pre><pre>S3_BUCKET_NAME,</pre><pre>‘demo.txt’,</pre><pre>args={‘ServerSideEncryption’: ‘AES256’}</pre><pre>)</pre><h4>Getting a list of files from S3 Bucket:</h4><p>The most convenient method to get a list of files from S3 Bucket using Boto3 is to use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Bucket.objects">S3Bucket.objects.all()</a> method:</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_bucket = s3_resource.Bucket(S3_BUCKET_NAME)</pre><pre>print(‘Listing Amazon S3 Bucket objects/files:’)</pre><pre>for obj in s3_bucket.objects.all():</pre><pre> print(f’ — {obj.key}’)</pre><p>Otherwise, we can use tlist_objects_v2()</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>s3_bucket = s3_client.list_objects_v2(Bucket=S3_BUCKET_NAME)</pre><pre>print(‘Listing Amazon S3 Bucket objects/files:’)</pre><pre>for obj in s3_bucket.objects.all():</pre><pre> print(f’ — {obj.key}’)</pre><h4>Filtering results of S3 list operation using Boto3 :</h4><p>If you need to get a list of S3 objects whose keys are starting from the specific prefix, you can use the .filter() method to do this:</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_bucket = s3_resource.Bucket(S3_BUCKET_NAME)</pre><pre>print(‘Listing Amazon S3 Bucket objects/files:’)</pre><pre>for obj in s3_bucket.objects.filter(Prefix=’demo’):</pre><pre> print(f’ — {obj.key}’)</pre><h4>Downloading file object from S3 Bucket :</h4><p>You can use the download_file() method to download the S3 object to your local file system:</p><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_object = s3_resource.Object(S3_BUCKET_NAME, ‘demo.txt’)</pre><pre>s3_object.download_file(‘/tmp/demo.txt’)</pre><pre>print(‘S3 object download complete’)</pre><h4>Reading files from the S3 bucket into memory :</h4><pre>#!/usr/bin/env python3</pre><pre>import io</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_object = s3_resource.Object(S3_BUCKET_NAME, ‘demo.txt’)</pre><pre>with io.BytesIO() as f:</pre><pre>s3_object.download_fileobj(f)</pre><pre>f.seek(0)</pre><pre>print(f’Downloaded content:\n{f.read()}’)</pre><h4>Deleting S3 objects using Boto3 :</h4><p>To delete an object from Amazon S3 Bucket, you need to call the delete() method of the object instance representing that object:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>s3_object = s3_resource.Object(bucket_name, ‘new_demo.txt’)</pre><pre>s3_object.delete()</pre><pre>print(‘S3 object deleted’)</pre><h4>Renaming S3 file object using Boto3:</h4><p>There’s no single API call to rename an S3 object. So, to rename an S3 object, you need to copy it to a new object with a new name and then deleted the old object:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>def rename_s3_object(bucket_name, old_name, new_name):</pre><pre> old_s3_object = s3_resource.Object(bucket_name, old_name)</pre><pre> new_s3_object = s3_resource.Object(bucket_name, new_name)</pre><pre> new_s3_object.copy_from(</pre><pre> CopySource=f’{bucket_name}/{old_name}’)</pre><pre> old_s3_object.delete()</pre><pre> print(f’{bucket_name}/{old_name} -&gt; {bucket_name}/{new_name}’)</pre><pre>rename_s3_object(S3_BUCKET_NAME, ‘demo.txt’, ‘new_demo.txt’)</pre><h4>Copying file objects between S3 buckets using Boto3:</h4><p>To copy file objects between S3 buckets using Boto3, you can use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Object.copy_from">copy_from()</a> method.</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>BUCKET_NAME = “demo-bucket”</pre><pre>def copy_object(bucket, src_object, dst_object):</pre><pre> s3_resource = boto3.resource(‘s3’, region_name=AWS_REGION)</pre><pre> s3_resource.Object(bucket, dst_object).copy_from(</pre><pre> CopySource=f’{bucket}/{src_object}’)</pre><pre>copy_object(bucket=BUCKET_NAME, src_object=’demo1.txt’, dst_object=’demo2.txt’)</pre><h4>Creating S3 Bucket Policy using Boto3 :</h4><p>To specify requirements, conditions, or restrictions for accessing the Amazon S3 Bucket, you have to use Amazon S3 Bucket Policies</p><p>Let’s use the Boto3 library to set up this policy in the S3 bucket:</p><pre>#!/usr/bin/env python3</pre><pre>import json</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>BUCKET_POLICY = {</pre><pre> “Version”: “2012–10–17”,</pre><pre> “Statement”: [</pre><pre>  {</pre><pre>  “Principal”: {</pre><pre>  “AWS”: “*”</pre><pre>  },</pre><pre>  “Action”: [</pre><pre>  “s3:*”</pre><pre> ],</pre><pre> “Resource”: [</pre><pre>  f”arn:aws:s3:::{S3_BUCKET_NAME}/*”,</pre><pre>  f”arn:aws:s3:::{S3_BUCKET_NAME}”</pre><pre> ],</pre><pre> “Effect”: “Deny”,</pre><pre> “Condition”: {</pre><pre>  “Bool”: {</pre><pre>   “aws:SecureTransport”: “false”</pre><pre>  },</pre><pre>  “NumericLessThan”: {</pre><pre>   “s3:TlsVersion”: 1.2 </pre><pre>     }</pre><pre>   }</pre><pre>  }</pre><pre> ]</pre><pre> }</pre><pre> policy_document = json.dumps(BUCKET_POLICY)</pre><pre>s3_client.put_bucket_policy(Bucket=S3_BUCKET_NAME,  Policy=policy_document)</pre><pre>print(‘Bucket Policy has been set up’)</pre><h4>Deleting S3 Bucket Policy using Boto3 :</h4><p>To delete the S3 Bucket Policy, you can use the delete_bucket_policy() method of the S3 client:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-2”</pre><pre>S3_BUCKET_NAME = “hands-on-cloud-demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>s3_client.delete_bucket_policy(Bucket=S3_BUCKET_NAME)</pre><pre>print(‘Bucket Policy has been deleted’)</pre><h4>Generating S3 presigned URL using Boto3:</h4><p>If you need to share files from a non-public Amazon S3 Bucket without granting access to AWS APIs to the final user, you can create a <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.generate_presigned_url">pre-signed URL</a> to the Bucket Object:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_client = boto3.client(“s3”, region_name=AWS_REGION)</pre><pre>def gen_signed_url(https://medium.com/bucket_name,%20object_name):</pre><pre> url = s3_client.generate_presigned_url(https://medium.com/ClientMethod=%E2%80%99get_object%E2%80%99,%3C/pre%3E%3Cpre%3E%20Params=%7B%E2%80%98Bucket%E2%80%99:%20bucket_name,%20%E2%80%98Key%E2%80%99:%20object_name%7D,ExpiresIn=3600)</pre><pre> print(url)</pre><pre>gen_signed_url(https://medium.com/S3_BUCKET_NAME,%20%E2%80%98demo.txt%E2%80%99)</pre><p>The S3 client’s generate_presigned_url() method accepts the following parameters:</p><ul><li>ClientMethod (<em>string</em>) — The Boto3 S3 client method to presign for</li><li>Params (<em>dict</em>) — The parameters need to be passed to the ClientMethod</li><li>ExpiresIn (<em>int</em>) — The number of seconds the presigned URL is valid for. By default, the presigned URL expires in an hour (3600 seconds)</li><li>HttpMethod (<em>string</em>) — The HTTP method to use for the generated URL. By default, the HTTP method is whatever is used in the method’s model</li></ul><h4>Enabling S3 Bucket versioning using Boto3 :</h4><p>S3 Bucket versioning allows you to keep track of the S3 Bucket object’s versions over time. Also, it safeguards against accidental object deletion. Boto3 will retrieve the most recent version of a versioned object on request. When a new version of an object is added, the object takes up the size of storage of the versions added together; i.e., a 2MB file with 5 versions will take up 10MB of space in the storage. To enable versioning for the S3 Bucket, you need to use the enable_version() method:</p><pre>#!/usr/bin/env python3</pre><pre>import boto3</pre><pre>AWS_REGION = “us-east-1”</pre><pre>S3_BUCKET_NAME = “demo-bucket”</pre><pre>s3_resource = boto3.resource(“s3”, region_name=AWS_REGION)</pre><pre>def enable_version(bucket_name):</pre><pre> versioning = s3_resource.BucketVersioning(bucket_name)</pre><pre> versioning.enable()</pre><pre> print(f’S3 Bucket versioning: {versioning.status}’)</pre><pre>enable_version(S3_BUCKET_NAME)</pre><h3>Conclusion :</h3><p>One of the things I always wished I knew before working on S3 using Boto3 is that S3 is object storage, it doesn’t have a real directory structure and The “/” is rather cosmetic that is used to simulate a simple file system and hence S3 objects cannot have “/” in their name. If you wish to explore more functionalities of Boto3 for S3 check this <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html">doc</a>. And I guess that’s all for now. !HAPPY-CODING!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df58e7038175" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/control-aws-s3-using-boto3-df58e7038175">Control AWS S3 using Boto3</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boto3: AWS SDK for Python]]></title>
            <link>https://medium.com/featurepreneur/boto3-aws-sdk-for-python-e7391b9901c5?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7391b9901c5</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-boto-boto3]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[boto3-python]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Sat, 23 Jul 2022 21:01:49 GMT</pubDate>
            <atom:updated>2022-08-04T13:30:39.242Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction :</h4><p>By now, it’s well-known that AWS(Amazon Web Services) is the biggest cloud computing platform. Even though Microsoft Azure and other platforms are catching up slowly, AWS will not be dethroned anytime soon. AWS can be controlled using many ways like through the AWS Console, AWS CLI, Python, Java, and other programming languages.</p><p>Boto3 is the AWS SDK(Software Development Kit) for Python to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services. Boto3 is maintained and published by AWS. Boto (pronounced boh-toh) was named after the freshwater dolphin native to the Amazon river. The name was chosen by the author of the original Boto library, Mitch Garnaat, as a reference to the company.</p><p>We can find the latest documentation at the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">Boto3 doc site</a>, including a list of services that Boto3 supports, and its source code <a href="https://github.com/boto/boto3">here</a></p><h4>Boto3 Usage :</h4><p>The Boto3 library allows us to easily integrate our Python applications, libraries, or script with AWS services and gives us full control over all AWS services like Amazon EC2, Amazon VPC, Amazon S3, Amazon DynamoDB, and others. Although, we need to keep in mind that the user or service making the API calls must have the actual permissions to call AWS APIs. Those permissions are controlled by the AWS IAM service, defined in IAM Policy, and granted to the user or service through the IAM Role.</p><p>To start using the Boto3 library to interact with AWS APIs, we have to install the Python boto3 module, import it from the Python program code, and use boto3.client() or boto3.resource() method to connect to a specific AWS service API endpoint. Finally, we would have to use the Boto3 client or resource object methods to interact with service-specific API.</p><h4>Installation :</h4><p>If your computer does not already have Boto3 or AWS tools, you should install them first. There are two Python modules we need to be able to manage AWS services from the command-line (CLI) and your Python scripts:</p><ul><li>boto3 — the AWS SDK for Python</li><li>awscli — command-line tools allowing you to manage AWS services from your terminal</li></ul><p>We need to install the AWS CLI tools to configure the AWS environment on your machine.</p><p>To install Boto3 on your computer, go to your terminal and run the following command:</p><pre>pip install boto3</pre><p>We have the SDK, but we won’t be able to use it right away. This is because the SDK doesn’t know which AWS account it should connect to.</p><p>To make it run your AWS account, you’ll need to provide valid AWS credentials. If you already have an IAM user with full permissions to S3, we can use those users’ credentials (their access key and their secret access key) without needing to create a new user. Otherwise, create a new AWS user and then store the new credentials.</p><p>To install the AWS CLI tools, you have to run another command in your terminal:</p><pre>pip install awscli</pre><h4>Configuring AWS environment :</h4><p>AWS CLI is a set of command-line tools for accessing AWS from the terminal shell. Those tools are available for us through the `aws` command. We will use the subcommand `configure` to set up an AWS environment on your workstation. To configure the AWS environment, type the following command in your terminal:</p><pre>aws configure</pre><p>This command will walk you through an environment configuration process and ask you for 4 things:</p><ul><li>AWS Access Key</li><li>AWS Secret Access Key</li><li>Default region name –&gt; enter us-east-1</li><li>Default output format –&gt; enter json</li></ul><p>The `aws configure` command updates a text file that the awscli, boto3, or any other software that interacts with AWS. If you are on macOS or Linux user, the aws configure command updates the text file at this path:</p><pre>~/.aws/credentials</pre><p><strong>**NOTE: </strong><em>The aws configure tool allows you not to store your AWS credentials (the AWS Access and Secret Keys) in your Python scripts.</em></p><h4>Passing AWS credentials to Boto3 Client Directly :</h4><p>In cases where we cannot use an external tool like AWS CLI, we can directly use them in the client() as parameters. To pass AWS credentials to the Boto3 client, you have to provide them in the aws_access_key_id and aws_secret_access_key variables, region_name variable for example:</p><pre>import boto3</pre><pre>client = boto3.client(</pre><pre>‘ec2’,</pre><pre>aws_access_key_id=”XXXXXXX”,</pre><pre>aws_secret_access_key=”YYYYYYY”,</pre><pre>region_name=”us-east-1&quot;</pre><pre>)</pre><p><strong>**NOTE: </strong><em>Storing your AWS credentials in your scripts is not secure and, you should never do this, we can set them as environment variables or use `.env` file and load it into the Python script but even storing AWS Access and Secret Keys in a plain text file is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.</em></p><h4>Testing AWS credentials via AWS CLI :</h4><p>A common way of testing AWS credentials is by running the following command:</p><pre>aws sts get-caller-identity | tee</pre><p>This command will produce information about your AWS Account and User IDs. The output will look like this:</p><pre>{</pre><pre>“UserId”: “AKIAYQV4J5VM5FWGJMGB”,</pre><pre>“Account”: “012345678901”,</pre><pre>“Arn”: “arn:aws:iam::012345678901:user/admin”</pre><pre>}</pre><h4>Testing AWS credentials via Boto3 :</h4><p>We can request the same information by using the Boto3 library. Here’s an example of a Python script to do that:</p><pre>#!/usr/bin/env python3</pre><pre>import json</pre><pre>import boto3</pre><pre>client = boto3.client(‘sts’)</pre><pre>response = client.get_caller_identity()</pre><pre>user_id = response[‘UserId’]</pre><pre>account = response[‘Account’]</pre><pre>arn = response[‘Arn’]</pre><pre>output = {</pre><pre>`‘UserId’: user_id,</pre><pre>‘Account’: account,</pre><pre>‘Arn’: arn</pre><pre>}</pre><pre>print(json.dumps(output, indent=4))</pre><p>In the Python code above, we did :</p><ul><li>Import of a built-in json module to process JSON response from theget_caller_identity() method — (<strong>line 3</strong>)</li><li>Import of the boto3 module to have an ability to make calls to AWS APIs — (<strong>line 4</strong>)</li><li>Instantiate the AWS Security Token Service (STS) client — (<strong>line 6</strong>)</li><li>Make a call to the AWS STS service to get information about AWS Account and User IDs and save the service response to the variable — (<strong>line 8</strong>)</li></ul><p><strong>**NOTE: </strong><em>All AWS API calls at the Boto3 library return response information in the form of a Python dictionary.</em></p><p>Now, run the Python script to get the same output as you got from the AWS CLI call</p><h4>Botocore :</h4><p>The AWS CLI and Boto3 library are built on top of a standard <a href="https://github.com/boto/botocore">botocore</a> module. This module is a low-level Python library that takes care of low-level operations required to send secure HTTPS API requests to AWS and respond. The botocore module is responsible for:</p><ul><li>Configuring and handling AWS user sessions and credentials during API calls</li><li>Providing a fine-granular interface to manage low-level AWS API calls to a specific service (for example, AWS STS <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html">GetCallerIdentity</a> API call)</li><li>Serializing and deserializing input parameters, and XML responses received from API calls into Python dictionaries</li><li>Providing low-level clients and high-level resources abstractions that we’re using to interact with AWS services from Python</li></ul><p>In short, botocore is a utility module that allows you to forget about dealing with really low-level API calls to AWS services and use more high-level methods to interact with them.</p><h4>Client Vs Resource :</h4><p>At its core, all that Boto3 does is call AWS APIs on your behalf. For the majority of the AWS services, Boto3 offers two distinct ways of accessing these abstracted APIs:</p><ul><li>Client<strong>:</strong> low-level service access</li><li>Resource<strong>:</strong> higher-level object-oriented service access</li></ul><p>You can use either to interact with S3. We started using the Boto3 client to make an API call to the AWS STS service. At the same time, you may find lots of examples on the internet, where people are using the client() instead of the resource() method. So, what are the differences?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*B5lpf1fwozAF4s_2Au9LNQ.png" /></figure><p>The major differences between boto3.client() and boto3.resource():</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*cQI6QVCrUqa9yXiMD0mz8g.png" /></figure><p>In summary, using the resource() allows us to write a more readable code and avoid processing response dictionaries. In addition to that, it allows you to lazily iterate over a large number of returned objects without thinking about pagination or memory utilization.</p><h4>Listing IAM Roles using Client or Resource :</h4><p>As an example for comparing both APIs, let’s get all IAM Roles from the AWS account using the client():</p><pre>import boto3</pre><pre>client = boto3.client(‘iam’)</pre><pre>paginator = client.get_paginator(‘list_roles’)</pre><pre>aws_roles = []</pre><pre>for page in paginator.paginate():</pre><pre> for role in page[‘Roles’]:</pre><pre>  aws_roles.append(role[‘RoleName’])</pre><pre>print(‘\n’.join(aws_roles))</pre><p>Now, let’s do the same thing but with the help of resource() instead:</p><pre>import boto3</pre><pre>iam_resource = boto3.resource(‘iam’)</pre><pre>aws_roles = []</pre><pre>for role in iam_resource.roles.all():</pre><pre> aws_roles.append(role.name)</pre><pre>print(‘\n’.join(aws_roles))</pre><p>As you can see, the resource() simplifies our code by removing the need to use a paginator and additional nested for-loop.</p><p>We can also get access to the client() methods from the resource():</p><pre>iam_resource = boto3.resource(‘iam’)</pre><pre># getting access to the client from resource object</pre><pre>iam_client = iam_resource.meta.client</pre><pre>paginator = iam_client.get_paginator(‘list_roles’)</pre><pre>aws_roles = []</pre><pre>for page in paginator.paginate():</pre><pre> for role in page[‘Roles’]:</pre><pre>  aws_roles.append(role[‘RoleName’])</pre><pre>print(‘\n’.join(aws_roles))</pre><h4>Sessions in Boto3 :</h4><p>The boto3.Session() is a class that stores configuration state and allows you to create the Boto3 clients() and resources(). Dealing with multi-region and multi-account AWS API calls topic is out of the scope of this article. Still, I have to mention that it is preferable to use session when you’re working with multi-region or multi-account code:</p><pre>import boto3</pre><pre>session = boto3.Session()</pre><pre>s3 = session.client(‘s3’)</pre><pre>ddb = session.resource(‘dynamodb’)</pre><p>Using boto3.Session() becomes even more important when you need to assume roles across different AWS accounts.</p><h4>Assuming IAM role using Boto3 :</h4><p>To assume an IAM role using Boto3 you need to use the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#STS.Client.assume_role">assume_role()</a> method of the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts.html#client">STS client</a>.</p><p><strong>**NOTE:</strong> <em>you need to make sure, that your current user </em><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html"><em>has been granted permission to assume that role</em></a><em>.</em></p><pre>import boto3</pre><pre>CURRENT_ACCOUNT_SESSION = boto3.Session()</pre><pre>STS_CLIENT = CURRENT_ACCOUNT_SESSION.client(‘sts’)</pre><pre>AWS_ACCOUNT_ID = ‘585584209241’</pre><pre>ROLE_NAME = ‘demo-role’</pre><pre>assumed_role_object = STS_CLIENT.assume_role(</pre><pre>RoleArn=f’arn:aws:iam::{AWS_ACCOUNT_ID}:role/{ROLE_NAME}’,</pre><pre>RoleSessionName=f’{ROLE_NAME}-Session’</pre><pre>)</pre><pre>assumed_role_credentials = assumed_role_object[‘Credentials’]</pre><pre>ASSUMED_ROLE_SESSION = boto3.Session(</pre><pre>aws_access_key_id=assumed_role_credentials[‘AccessKeyId’],</pre><pre>aws_secret_access_key=assumed_role_credentials[‘SecretAccessKey’],</pre><pre>aws_session_token=assumed_role_credentials[‘SessionToken’],</pre><pre>)</pre><pre>instances = ASSUMED_ROLE_SESSION.resource(‘ec2’, region_name=”us-east-1&quot;).instances.all()</pre><pre>for instance in instances:</pre><pre>print(f’Instance ID: {instance.id}’)</pre><h4>Conclusion :</h4><p>At this point, we’re done! It would be best if you now had a basic understanding of Boto3 with the examples given here. May this tutorial be a stepping stone in your journey to building something great using AWS!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7391b9901c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/boto3-aws-sdk-for-python-e7391b9901c5">Boto3: AWS SDK for Python</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Keep your Secrets off your code: GitGuardian]]></title>
            <link>https://medium.com/featurepreneur/keep-your-secrets-off-your-code-gitguardian-509ade1084d4?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/509ade1084d4</guid>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[security-services]]></category>
            <category><![CDATA[gitguardian]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Mon, 11 Jul 2022 17:52:03 GMT</pubDate>
            <atom:updated>2022-07-18T23:48:18.900Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction :-</h4><p>Everyone tends to make mistakes, and that’s just part of the learning experience. But what if that mistake is something with grave consequences, like leaking essential credentials in your code which could compromise your entire organization to hackers. Nowadays, security and privacy get invaded by such leaks instantly due to bots. These risks could be avoided by using GitGuardian.</p><p>GitGuardian is a global cybersecurity startup focusing on code security solutions for DevOps generation. They are a leader in the market of secrets detection and remediation. Their solutions are used by thousands of developers in all industries. GitGuardian helps developers, cloud operation, security, and compliance professionals secure software development and define and enforce policies consistently and globally across all their systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*P5RHKefcHcuCckQr" /></figure><h4>Advantages of GitGuardian :-</h4><ul><li>GitGuardian solutions monitor public and private repositories in real-time, detect secrets, and alert to allow investigation and quick remediation.</li><li>Secure your software development lifecycle with enterprise-grade secrets detection.</li><li>Eliminate blind spots with an automated, battle-tested detection engine.</li><li>Build custom detectors to enhance your scans for secrets unique to your organization.</li><li>Discover vulnerabilities early and collaboratively, then harness rapid remediation to save time, money, and paperwork.</li><li>Eliminate secrets exposed on internal repositories, GitGuardian Internal Repository Monitoring focuses exclusively on your organization’s repositories.</li><li>With pre-commit git hooks, developers can scan changes before pushing their code and keep secrets out of the VCS.</li><li>Decentralize and automate incident response by alerting the developers involved, and Collecting feedback from the field to understand how the incident interacts with other services and software components.</li></ul><h3>How to activate GitGuardian :-</h3><ol><li>Go to <a href="https://www.gitguardian.com/">GitGuardian</a> site</li><li>Click “Start for Free” button</li><li>Sign up with your GitHub Account</li><li>Authorize GitGuardian to access all your repositories</li><li>In the GitGuardian Dashboard, check if all your repositories are present and click the “Launch Scan” button</li></ol><p>Congratulations, your repositories are now under the real-time protection of GitGuardian.</p><h3>Git hooks integrations using GitGuardian :-</h3><p>It is essential to have secrets detection within your git server or remote git host because this is where the active threats live. But, secrets that end up within these git repositories must be considered compromised, and revoking them can take time and resources. It is always best to capture secrets before your repository is compromised. This can be achieved with a <strong>pre-commit git hook</strong>. For more information check this <a href="https://blog.gitguardian.com/setting-up-a-pre-commit-git-hook-with-gitguardian-shield-to-scan-for-secrets/">tutorial</a>.</p><p>Similarly, we can also integrate with <a href="https://www.youtube.com/watch?v=uc70CE1MXvM">pre-push hook</a> and <a href="https://docs.gitguardian.com/internal-repositories-monitoring/integrations/git_hooks/pre_receive">pre-receive hook</a></p><p><strong>**NOTE: </strong><em>Remember that these implementations require GitGuardian API calls, which allows for only 1000 calls per month in the free plan. Upgrade to Business plan to use up to 10k calls per month</em></p><h3>Set up Alerts using GitGuardian :-</h3><p>Other than email alerts, we can integrate GitGuardian and set up notifications for incidents with various other popular communication services like Discord, Slack, Jira, etc. We can do this by:</p><ol><li>Go to <a href="http://dashboard.gitguardian.com/">GitGuardian Dashboard</a></li><li>Choose “Integrations” option in the sidebar</li><li>Scroll to the bottom of the page</li><li>Under “Alerting” tab, press the install button on which app you would like to use</li><li>Follow the instructions given on the page and submit</li></ol><h3>Conclusion :-</h3><p>GitGuardian is a startup that is respected by the tech community due to its favourability for individual developers and small-scale organizations, and its open-source code. Aside from that. They are the best in their respective field and will keep getting better due to the frequent updates and improvements. If you are not comfortable with GitGuardian or if you feel that it’s not the right one, there are plenty of alternatives that are listed on the <a href="https://www.gitguardian.com/gitguardian-alternatives">GitGuardian site</a>.</p><p>! <strong>Code safe </strong>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=509ade1084d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/keep-your-secrets-off-your-code-gitguardian-509ade1084d4">Keep your Secrets off your code: GitGuardian</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pre-Requisites for Installing Ubuntu]]></title>
            <link>https://medium.com/featurepreneur/pre-requisites-for-installing-ubuntu-1b593c5ef7ab?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b593c5ef7ab</guid>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[dual-boot]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[ubuntu-desktop]]></category>
            <category><![CDATA[backup]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Sat, 25 Jun 2022 19:19:21 GMT</pubDate>
            <atom:updated>2022-08-11T10:03:58.726Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>Ubuntu is a very popular Linux Distros, and many Windows OS users are either switching to Ubuntu or dual-booting it alongside Windows. They can be difficult for first-timers, and even experienced users can make a mistake while doing this. So in this article, we will see steps to avoid errors during and after installation.</p><h3>Step — 1 — Check Requirements:</h3><p>First check the requirements for the Ubuntu version you are going to install, if not choose a different version or alternative.</p><h3>Step — 2 — Make a separate new hard disk partition for installing Ubuntu:</h3><p>While it’s also possible to do this during installation, I would recommend everyone to make the disk partition for Ubuntu beforehand.</p><blockquote>Learn how to create a disk partition in Windows, by following this <a href="https://support.microsoft.com/en-us/windows/create-and-format-a-hard-disk-partition-bbb8e185-1bda-ecd1-3465-c9728f7d7d2e">article</a>.</blockquote><h3>Step — 3 — Create a backup:</h3><p>I would strongly suggest everyone create a backup in the cloud storage or external storage for at least the essential files and sensitive data like photos, videos, documents, contacts, and messages, in case of some failure.</p><h3>Step — 4 — Create a log</h3><p>Create a log with all the Technical information of the PC with Device Specifications, Machine Information, Drivers Information, Windows Specifications including Product key, and Storage Status. This will help identify errors and recover from them.</p><h3>Step — 5 — Disable Secure Boot</h3><p>Secure Boot helps to make sure that your PC boots using only firmware that is trusted by the manufacturer which usually only supports OS Microsoft Windows 8.1 and higher. And you have to disable Secure Boot to run some PC graphics cards, hardware, or operating systems such as Linux or the previous version of Windows.</p><blockquote>So how to disable Secure Boot? — All you have to do is to open the PC BIOS menu (access this menu by pressing a key during the bootup sequence, such as F1, F2, F12, or Esc — different for different manufacturers) and find the Secure Boot setting and set it to Disabled.</blockquote><blockquote>If you using Windows 8.1 or Windows 10, then you can disable Secure Boot without entering BIOS. You just have to hold the Shift key while selecting Restart. Go to Troubleshoot &gt; Advanced Options: UEFI Firmware Settings and choose to disable the option for Secure Boot settings.</blockquote><p><strong>**NOTE:</strong> <em>In some cases, you may need to change other settings in the firmware, such as enabling a Compatibility Support Module (CSM) to disable Secure Boot. To use a CSM, you may also need to reformat the hard drive using the Master Boot Record (MBR) format, and then reinstall Windows.</em></p><h3>Step — 6 — Disable Fast Startup</h3><p>The majority of users forget to disable Fast Boot or Fast Startup before dual booting Linux with Windows — which results in the system booting into Windows after Linux installation without showing the GRUB menu. Fast Startup is a setting that helps your PC start up faster after shutdown and by default, it is enabled in Windows.</p><blockquote>How to disable Fast Startup? — Unlike Secure Boot you don’t have to enter BIOS to disable Fast Startup you can do it by going into Power Options in Control Panel as follows :</blockquote><blockquote>Open Control Panel change view by “Large icons/Small icons”</blockquote><blockquote>Click Power Options.</blockquote><blockquote>Click Choose what the power buttons do.</blockquote><blockquote>Click Change settings that are currently unavailable.</blockquote><blockquote>Scroll down to Shutdown settings and uncheck Turn on fast startup.</blockquote><blockquote>Click Save changes.</blockquote><h3>Step — 7 — Disable RST</h3><p>Intel Rapid Storage Technology (RST) is a solution built into a range of Intel chipsets. On platforms that have RST support built and enabled in the computer’s BIOS, it allows users to group and manage multiple hard disks as single volumes.</p><p>If you intend to install Ubuntu on a computer that supports RST functionality, you may need to make operational adjustments to your setup before you can proceed. By default, the Ubuntu installer can detect certain RAID configurations, but it may not necessarily be able to access and use the hard disks grouped in them. If the Ubuntu installer cannot detect the hard disks you need, then before you can install Ubuntu, you will need to turn off RST in the computer’s BIOS. The exact terminology and steps required to access and manage RST in BIOS often depend on the specific implementation by the platform vendor.</p><blockquote>Follow this Guide for <a href="https://discourse.ubuntu.com/t/ubuntu-installation-on-computers-with-intel-r-rst-enabled/15347">Ubuntu installation on computers with Intel(R) RST enabled</a>.</blockquote><h3>Step — 9— Test the trial beforehand</h3><p>If you are doubtful about Ubuntu installation you can try it before installing using the “Try Ubuntu” option. While it may not be able to exactly replicate the complete functionality, you will be able to test the core features in this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/0*__3kXOC97HgHqPfj" /></figure><h3>Step — 8 — Patience and Self-Control</h3><p>Patiently note down all the steps you do, and configurations you choose and don’t press any key unnecessarily during the installation.</p><p>Never make any decision without comprehending its implications, and don’t hesitate to look up any unknown options, since there are constant changes to the installation process it’s hard even for professionals to keep up.</p><p>If you are impatient or lazy about this you may regret it.</p><blockquote>**<strong>Note</strong> : It&#39;s also important to disable bitlocker if you have enabled it in your PC as it may completely block the installation of other OS for security purposes</blockquote><h3>Conclusion:</h3><p>The steps I have mentioned are something I practice when installing Ubuntu for my PC or others. They are just some of the pre-cautionary and recovery measures I know from my experience and there may be a lot more than these. While these steps may feel overly cynical, it’s always better to be safe than sorry.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b593c5ef7ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/pre-requisites-for-installing-ubuntu-1b593c5ef7ab">Pre-Requisites for Installing Ubuntu</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GeoPy]]></title>
            <link>https://medium.com/featurepreneur/geopy-dfc65e40f4c9?source=rss-fb89ea19ee2------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfc65e40f4c9</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[geocoding]]></category>
            <category><![CDATA[geopy]]></category>
            <dc:creator><![CDATA[SivaraamTK]]></dc:creator>
            <pubDate>Sat, 25 Jun 2022 16:10:15 GMT</pubDate>
            <atom:updated>2022-08-04T13:12:19.184Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction:</h3><p>GeoPy is a Python client for several popular geocoding web services. GeoPy makes it easy for Python developers to locate the coordinates of addresses, cities, countries, and landmarks across the globe using third-party geocoders and other data sources.</p><p>We can get information such as postal code (zip code), city, neighbourhood, etc. GeoPy can make API requests to Nominatim (a geocoding software used for OpenStreetMap), Google Maps, Bing Maps, and many others.</p><p>One thing to note is that GeoPy does not provide this info, but it simply helps the user connects to them, so users need to understand the terms, pricing, and limitations of each API. Nominatim, famous for small-scale projects, is a free option that allows a limited number of requests. Please see <a href="https://operations.osmfoundation.org/policies/nominatim/">here</a> for Nominatim’s API use policy. If you need to use other APIs, you will likely need to obtain an API key directly from the API providers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*oRpbdehGt6bNNxXxXrKP4g.jpeg" /></figure><h3>Installing Geopy:</h3><p>The first thing to do is to install Geopy.</p><pre><em>pip install geopy</em></pre><h3>Establishing Connection:</h3><p>We need to establish a connection to APIs by setting up the geocoder. First import the geocoder you want to use, and initiate it.</p><pre><em>from geopy.geocoders import Nominatim</em></pre><pre><em>geocoder = Nominatim(user_agent = “&lt;specify_your_app_name_here&gt;”)</em></pre><p>For other APIs, you may need to put in other parameters such as username/password, API keys, etc. Please refer to the <a href="https://geopy.readthedocs.io/en">document</a>.</p><h3>Getting Latitude and Longitude:</h3><p>The general use of the geocoder is to get the actual coordinates from the address.</p><pre># after initiating geocoder</pre><pre><em>location = geocode(address)</em></pre><pre># returns location object with longitude, latitude and altitude instances</pre><pre>(location.latitude, location.longitude)</pre><h3>Getting Address Information:</h3><p>We can retrieve information about a location using the coordinates by reversing the geocoder</p><pre><em>location = geocode.reverse((lat, long))</em></pre><pre># returns geopy Location object</pre><p>Location objects have instances of address, altitude, latitude, longitude, and point. address returns a concatenated string of all information that belongs to the address.</p><h3>Getting Distance:</h3><p>We can also extract the distance between different locations from the coordinates. GeoPy offers both the great-circle distance and the geodesic distance.</p><pre><em>from geopy.distance import geodesic, distance</em></pre><pre><em>distance1_in_km = geodesic(coordinate1, coordinate2).km</em></pre><pre><em>distance2_in_km = distance.distance(coordinate1, coordinate2).km</em></pre><pre># note: coordinates must be in a tuple of (lat, long)</pre><h3>Sample Programs:</h3><h4>To geolocate a query to an address and coordinates:-</h4><pre><em>from geopy.geocoders import Nominatim</em></pre><pre><em>geolocator = Nominatim(user_agent=”getLoc”)</em></pre><pre><em>location = geolocator.geocode(“175 5th Avenue NYC”)</em></pre><pre><em>print(location.address)</em></pre><pre># Flatiron Building, 175, 5th Avenue, Flatiron, New York, NYC, New York, …</pre><pre><em>print((location.latitude, location.longitude))</em></pre><pre># (40.7410861, -73.9896297241625)</pre><pre><em>print(location.raw)</em> # Used to get raw dictionary with location information</pre><pre># {‘place_id’: ‘9167009604’, ‘type’: ‘attraction’, …}</pre><h4>To find the address corresponding to a set of coordinates:-</h4><pre><em>from geopy.geocoders import Nominatim</em></pre><pre><em>geolocator = Nominatim(user_agent=”getAddr”)</em></pre><pre><em>location = geolocator.reverse(“52.509669, 13.376294”)</em></pre><pre><em>print(location.address)</em></pre><pre># Potsdamer Platz, Mitte, Berlin, 10117, Deutschland, European Union</pre><pre><em>print((location.latitude, location.longitude))</em></pre><pre># (52.5094982, 13.3765983)</pre><pre><em>print(location.raw)</em></pre><pre># {‘place_id’: ‘654513’, ‘osm_type’: ‘node’, …}</pre><h4>To find the distance between two set of coordinates:-</h4><pre><em>from geopy import distance<br>newport_ri = (41.49008, -71.312796)<br>cleveland_oh = (41.499498, -81.695391)<br>print(distance.distance(newport_ri, cleveland_oh).miles)<br></em># 538.39044536<br><em>print(distance.great_circle(newport_ri, cleveland_oh).miles)</em><br># 536.997990696</pre><h3>Conclusion:</h3><p>Even though GeoPy may not seem to have much relevance to you, coordinates may add a lot more information when it is categorized appropriately and GeoPy can convert coordinates into other variables that can be more useful, and also has a direct relationship to the demographics. Once it is grouped into countries, states, cities, and neighbourhoods, we can see different trends across locations. It’s also advisable to try visualizing this information to get better insights into such transformations.</p><p>Check out <a href="https://geopy.readthedocs.io/">GeoPy Documentation</a> to see various usages of Geocoder with different geolocation services and more information on GeoPy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfc65e40f4c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/featurepreneur/geopy-dfc65e40f4c9">GeoPy</a> was originally published in <a href="https://medium.com/featurepreneur">featurepreneur</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>