<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ankit Ojha on Medium]]></title>
        <description><![CDATA[Stories by Ankit Ojha on Medium]]></description>
        <link>https://medium.com/@theankitojha?source=rss-5c8f459a424d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bz2CmzJCoI0iu3Q79bdvQg.jpeg</url>
            <title>Stories by Ankit Ojha on Medium</title>
            <link>https://medium.com/@theankitojha?source=rss-5c8f459a424d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:30:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@theankitojha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Mastering Vi/ViM Editor]]></title>
            <link>https://theankitojha.medium.com/mastering-vi-vim-editor-42d1b886da90?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/42d1b886da90</guid>
            <category><![CDATA[text-editor]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[vim]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[vi]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Wed, 30 Aug 2023 09:18:00 GMT</pubDate>
            <atom:updated>2023-08-30T09:18:00.054Z</atom:updated>
            <content:encoded><![CDATA[<p>While working with CLIs, we always encounter textual data. And, how fast and efficiently we can work with those textual data plays a significant role in getting things done and saving us time.</p><p>In this article, we are going to learn about the ViM editor and how we can use this tool for editing texts (files) on Linux machines.</p><blockquote><strong>What is Vi/ViM?</strong></blockquote><p>Vi <strong>(Visual)</strong> is the most powerful and iconic text editor tool available on Linux Systems and ViM<strong> (Vi IMproved)</strong> is just a newer and more advanced form of Vi.</p><p>Unlike nano editor, we do not get any keyboard shortcuts provided right in the text editor with Vi, so it may seem daunting at first. However, it is guaranteed that using Vi/ViM editor will increase our efficiency and productivity while working with texts.</p><blockquote><strong>Different modes of Vi Editor</strong></blockquote><p>There are 4 different modes in Vi Editor and each of these modes carries a specific purpose. So, one must have a clear understanding of all these different modes for an efficient text editing experience.</p><ol><li>Command Mode</li><li>Insert Mode</li><li>Escape/Ex Mode</li><li>Visual Mode</li></ol><h4>Command Mode</h4><p>This is the default mode with Vi and it is active as soon as we open a file with Vi editor.</p><p>In this mode, we can only edit (delete/paste/copy) texts in the file but, cannot perform write operations. It may look like, we can just start typing, but we may get some unexpected behavior when we try to do so.</p><p>Some of the operations that we can perform in this mode will be:</p><pre>h : Move one character left<br>l : Move one character right<br><br>j : Move one line down (Alt: Enter)<br>k : Move one line up<br><br>O : Move to the beginning of the current line<br>$ : Move ot the end of the current line<br><br>gg  : Move to the top of the file<br> G  : Move to the bottom of the file<br><br>&#39;N&#39;G : Move to the N(th) line of the file<br><br>w : Move one word ahead<br>b : Move one word back<br><br>) : Move one sentence ahead<br>( : Move one sentence back<br><br>} : Move one paragraph ahead<br>{ : Move one paragraph back<br><br>b : Move to the beginning of the word<br>e : Move to the ending of the word<br><br>u : Undo previous change(s), if done in the file<br><br>yy : Copy the current line (yanking texts)<br> p : Paste the copied line<br><br>   dd : Delete the current line<br>&#39;N&#39;dd : Delete &#39;N&#39; number of lines below from the current line<br>   dw : Deletes a word<br>  Ndw : Deletes &#39;N&#39; words<br>    x : Delete a single character at the cursor<br><br>:set number : Show line number before each line in file (temporary)</pre><h4>Insert Mode</h4><p>This is the mode that allows us to insert texts into the file.</p><p>This mode is activated upon simply typing ‘i’ in command mode. Whatever we now type, the Vi editor will insert the characters rather than interpreting them as commands.</p><p>Now, there are various ways we can switch to this mode, rather than just using ‘i’. It depends on our use case. Some of the ways are:</p><pre>i : Insert text before the current cursor pointer<br>a : Insert text after the current cursor pointer<br><br>l : Insert text from beginning of the current line<br>A : Insert text from end of the current line<br><br>o : Open a new line below the current one<br>O : Open a new line above the current one<br><br>s : Substitute the current character<br>S : Substitute the current line<br><br>r : Replace the current character<br>R : Replace the current line</pre><h4>Escape Mode / Ex Mode</h4><p>This mode helps us perform tasks like saving files and executing commands.</p><p>It is activated by typing ‘:’ or ‘/’ when the Vi is in command mode. The cursor will then move to the last line of the screen and the vi editor waits for commands given by the user to perform respective actions.</p><p>Various operations that can be performed in this mode are:</p><pre><strong>Searching Texts:</strong><br>/text : Perform search in forward direction (Top-to-bottom)<br>?text : Perform search in reverse direction (Botton-to-Top)<br>    n : Search next occurance of the searched text in forward direction<br>    N : Search next occurance of the searched text in backward direction<br><br><strong>Save and Exit:</strong><br>   :w : Save the changes<br>  :wq : Save changes into the current file and exit (Alt command-&gt; :x)<br> :wq! : Forcefully save into the read-only file<br>  :q! : Exit without saving the changes<br><br><strong><em>NOTE:</em></strong> To save and exit we can also use -&gt; ZZ (capital Z) / Shift + zz (if caps lock is not ON)<br><br><strong>Replacing Texts with matching patterns:<br></strong>:s/&lt;exitstingWord&gt;/&lt;newWord&gt;/g : Replaces word(s) in the current line<br>:%s/&lt;existingWord&gt;/&lt;newWord&gt;/g : Replaces every word(s) in the file<br>:%s/&lt;existingWord&gt;/&lt;newWord&gt;/gc : Prompts before making substitutions, c is added at end<br><br><strong>Move/Copy existing lines to another line inside the document:<br></strong>:NcopyM : &#39;N&#39; refers to the line to be copied and &#39;M&#39; is the line number where it will be pasted<br>:NmoveM : same as above, except the line will be moved instead of copying</pre><blockquote><strong>Some more Tips and Tricks with Vi/ViM:</strong></blockquote><h4><strong>Password protect a file:</strong></h4><p>To password-protect a file, type :<strong>X </strong>(capital X) in the command mode and we will be asked to enter the desired encryption key.</p><p>Now, save and exit the file using the command-&gt; <strong>:wq</strong></p><p>Also, we can simply remove the encryption key by again typing <strong>:X </strong>in command mode and when prompted with the encryption key, leave it blank to remove the password. And, finally <strong>:wq</strong> to save the changes and exit.</p><h4>Recovering unsaved data of a file after a crash:</h4><p>Let’s suppose that we are trying to make changes to a file named ‘<strong>newfile</strong>’ and due to some unforeseen circumstances (like server crash), our file could not be saved properly.</p><p>Now, in order to recover our unsaved data of that file, we just need to edit the file named <strong>.&lt;fileName&gt;.swp.</strong></p><p>The ‘<strong>ls -al’ </strong>command will show the .swp file of your original file which was stored in the virtual memory (swap memory).</p><p>After editing the file, simply save it and we are good to go. Also, remove the old .<strong>filename.swp</strong> file after performing necessary operations.</p><h4>Invoke external binaries or external files within vim:</h4><p>Sometimes, we might need to insert the output of other commands directly in our file being edited.</p><p>Or, often me might encounter with situation where we need to append the contents of another file to the file that we are currently editing.</p><p>In such cases, we might use of this technique to get what we want.</p><pre>:r! &lt;binary&gt; : To invoke external binaries inside the file<br>:r &lt;/path/to/external/file&gt; : To copy the contents of external file to current file<br><br><strong>Examples:<br></strong>:r! netstat -tulpn | grep LISTEN<br>-&gt; This will print the output of that command in the current file<br><br>:r /etc/passwd<br>-&gt; This will copy the contents of /etc/passwd in the current file</pre><h4>Change letters, words or entire lines to UPPERCASE or lowercase:</h4><p>This is performed in the Ex mode of the vim.</p><pre><strong>NOTE: X </strong>refers to the number (or an integer value)<br><br><strong>For UPPERCASE<br>-------------<br>For letters:<br></strong>Position the cursor to the first letter, then type <strong>gUX</strong> and press right arrow key.<br><br><strong>For words:<br></strong>Place the cursor at the beginning of the word, then type <strong>gUXw<br><br>For entire line:<br></strong>Place the cursor anywhere in the line, then type <strong>gUU<br><br>NOTE:<br></strong>For<strong> lowercase, </strong>everything is same, except we should use &#39;u&#39; instead of &#39;U&#39; in each command.</pre><blockquote><strong>More about Vi/ViM:</strong></blockquote><p>To learn more about this tool, we can use following commands:</p><ol><li>man vim -&gt; This will display the <strong>man</strong>ual page for vim editor.</li><li>vimtutor -&gt; This is a built-in tutor for Vi/ViM editor in Linux systems.</li></ol><blockquote><strong>Conclusion</strong></blockquote><p>In this article, we talked a lot about Vi/ViM editor and how we can use it efficiently to work with textual data.</p><p>However, there are still many more tips and tricks to cover. Please feel free to share them in comments section below.</p><p>Thank you for your time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42d1b886da90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ChatGPT in Linux Terminal]]></title>
            <link>https://blog.gopenai.com/chatgpt-in-linux-terminal-4ba1e489dad8?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ba1e489dad8</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 11:02:10 GMT</pubDate>
            <atom:updated>2023-04-05T15:44:27.573Z</atom:updated>
            <content:encoded><![CDATA[<p>Wouldn’t it be amazing if we can bring up the power of chatGPT right into our Linux Terminal? Well, not to worry because that is what we are going to explore in this post.</p><p>To use ChatGPT from Linux Terminal, we need something called <strong>ShellGPT</strong>. Basically, it is a command-line tool that helps users to interact with ChatGPT from the Linux terminal.</p><p>Developed by Farkhod Sadykov (along with Eric and Loïc Coyle), ShellGPT is powered by OpenAI’s GPT large language model and it is capable of providing any sort of solutions that we want.</p><p>Now without any further talk, let’s get right into setting up ShellGPT.</p><h3><strong>Prerequisites</strong></h3><blockquote><strong>Install Python</strong></blockquote><p>Make sure you have python installed on your machine. This is because ShellGPT is built on top of python and it will require python to run.</p><pre>sudo apt update &amp;&amp; sudo apt upgrade -y<br>sudo apt install python3</pre><p>Once it is installed successfully, verify the installation as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*vQD6JFrUDjWx0r0o0syWiA.png" /></figure><blockquote><strong>Install Pip package Manager</strong></blockquote><p>Pip is a package manager for Python and we will be needing this to manage various packages and libraries (of Python) that are required for ShellGPT.</p><pre>sudo apt-get -y install python3-pip</pre><p>Again, verify the pip installation as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*brjpqFSlUKSBqNVaknolDg.png" /></figure><blockquote><strong>Setup venv module (Optional, but recommended)</strong></blockquote><p>This is because creating an isolated virtual environment in Linux can prevent any conflict with other libraries. To set up venv:</p><pre>sudo apt install python3-venv</pre><h3>Installation</h3><p>The installation process can be divided into 3 steps:</p><ol><li>Setting up environments</li><li>Getting OpenAI’s API Key for ChatGPT</li><li>Installing ShellGPT</li></ol><blockquote><strong>Setting up Environments</strong></blockquote><ol><li>Create a new directory and switch over to that newly created directory.</li></ol><pre>mkdir &lt;dirName&gt;<br>cd &lt;dirName&gt;<br><br>dirName: ANY NAME IS FINE (e.g: chatgpt-from-bash)</pre><p>2. Now create a virtual environment with the venv module as shown below:</p><pre>python3 -m venv &lt;virtual_environment_name&gt; (ANY NAME IS FINE)</pre><p>Example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yyya8wa2G0H97Rc7bkYa_A.png" /></figure><p>3. Finally, activate the virtual environment with:</p><pre>source &lt;virtual_environment_name&gt;/bin/activate</pre><p>Once, the virtual environment gets activated, you can notice that the Linux shell prompt changes as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-KlXT_rIPGGeTpfzMb5mOQ.png" /></figure><blockquote><strong>Getting OpenAI’s API Key for ChatGPT</strong></blockquote><p>So, to use the ChatGPT service from a Linux terminal, we will require an API Key of OpenAI.</p><p><strong><em>NOTE: </em></strong>Currently, OpenAI is offering $5 credits for trial use. Once the credit is exhausted, you need to pay for access to the API.</p><p>Getting back to the topic, let’s see how we can get the OpenAI API key for now.</p><ul><li>Login to <a href="https://platform.openai.com/">OpenAI’s platform</a>. If you don’t have an account, create one.</li><li>Once you get logged in successfully, click on the top right corner of the page (at your profile image) and select the “<strong>View API Keys</strong>” option.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*o6AScy4g7pzeMJy908xxLw.png" /></figure><p>You can see all your previously generated API keys (if any) on this page.</p><ul><li>Click on the “<strong>Create new Secret Key</strong>” option to generate a new API key.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*OLjMGzn5rGg5autLvp86uQ.png" /></figure><ul><li>A new pop-up box will open with your API key. Make sure to keep this key secret and private. <strong>Copy your API key </strong>in a secure file or location as you won’t be able to get that key after you click “OK”.</li><li>Finally, create an environment variable for this API Key in your Linux machine with the following command:</li></ul><pre>export OPENAI_API_KEY=&lt;your_OpenAI_API_key_here&gt;</pre><p>Verify the environment variable by running:</p><pre>env | grep OPENAI</pre><p>You can see your OpenAI API Key in the output if it is set correctly.</p><h4>NOTE:</h4><p>This method does not store the environment variable permanently. To do so, you need to add the following line in your <strong>.bashrc</strong> file:</p><pre>export OPENAI_API_KEY=&lt;your_OpenAI_API_key_here&gt;</pre><p>Once you add it, source your .bashrc file and you are good to go!</p><blockquote><strong>Installing ShellGPT</strong></blockquote><p>Once you are done with all the above steps, now it’s time to install ShellGPT on your machine and start exploring:</p><pre>pip3 install shell-gpt (IF YOU ARE IN A VIRTUAL ENVIRONMENT)<br>pip3 install shell-gpt --user (IF NOT IN A VIRTUAL ENVIRONMENT)</pre><p>That’s it. You have successfully integrated ChatGPT into your terminal.</p><h4>Examples using ShellGPT from Linux Terminal:</h4><blockquote><strong>For normal queries</strong></blockquote><p>Syntax: <strong>sgpt &quot;&lt;your_query&gt;&quot;</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oZAVS7s73F0Va8hSMGoYhw.png" /></figure><blockquote><strong>Chat Mode</strong></blockquote><p>Syntax: <strong>sgpt --chat &lt;unique_chat_session_name&gt; &lt;input_prompt&gt;</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yThPSdM9REzJhs6c-IoRyw.png" /></figure><blockquote><strong>Generate Code</strong></blockquote><p>Syntax: <strong>sgpt --code &quot;&lt;input_prompt&gt;&quot;</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eJxXXGwd7NueAvCSaQSTMA.png" /></figure><blockquote><strong>Generate and Execute Shell Commands</strong></blockquote><p>Syntax: <strong>sgpt --shell &quot;&lt;input_prompt&gt;&quot;</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KpQRN45ah2TZtXI6ReokKg.png" /></figure><p>That’s all. Happy Learning! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ba1e489dad8" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/chatgpt-in-linux-terminal-4ba1e489dad8">ChatGPT in Linux Terminal</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Swarm over Kubernetes as a container orchestration tool]]></title>
            <link>https://theankitojha.medium.com/docker-swarm-over-kubernetes-as-a-container-orchestration-tool-3186407f297a?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/3186407f297a</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[container-orchestration]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[swarm]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Fri, 02 Sep 2022 06:32:59 GMT</pubDate>
            <atom:updated>2022-09-02T06:32:59.296Z</atom:updated>
            <content:encoded><![CDATA[<p>As we know, the world is now running on containerization technology to ease the process of deploying and managing simple to complex applications.</p><p>They are very fast, portable, easy to manage and and scalable that any other traditional practices that we used to follow in early days. And, in order to scale or orchestrate our container infrastructure we need some <a href="https://www.ibm.com/cloud/learn/container-orchestration"><strong>container orchestration</strong></a><strong> </strong>tool like <strong>Docker Swarm </strong>or <strong>Kubernetes </strong>which help us to manage our containers.</p><p>We cannot neglect the fact that Kubernetes is the most widely used container orchestration tool right now. But, Docker Swarm comes with its own benefits.</p><p>I think that the container orchestration tool should be chosen depending on the complexity of our application. While Kubernetes is better at managing huge and very complex applications (thousands of deployments), Docker Swarm can be used if we need some lightweight solution and which can run up-to few hundred containers and dozens of nodes.</p><p>In this article, I am going to explain why and when to use <strong>Docker Swarm </strong>over <strong>Kubernetes </strong>as a container orchestration tool.</p><h4>Benefits of Docker Swarm over Kubernetes:</h4><p>Some of the benefits of Docker Swarm over Kubernetes are:</p><blockquote><strong>Easy to use:</strong></blockquote><p>In Docker Swarm, compose file is used to manage the environment. Hence, it is very easier to manage compose files that Kubernetes manifests.</p><blockquote><strong>Easy to setup:</strong></blockquote><p>Docker Swarm comes pre-installed with the installation of Docker whereas Kubernetes comes with many ways to install like <strong>minikube, kubespray, microk8s</strong> and many more.</p><blockquote><strong>Less resource usage:</strong></blockquote><p>Docker Swarm is best at using very few resource for running a production-grade cluster than Kubernetes.</p><p>While swarm is using few hundred MB&#39;s to get the master node up and running in smaller clusters, Kubernetes, in other hand has large overhead and needs few GB of RAM free to work.</p><blockquote><strong>Easy Administration:</strong></blockquote><p>The administration tooling in Docker Swarm is pretty easy and minimalist (like Portainer) that that of Kubernetes (like Rancher). Portainer can be integrated with Kubernetes as well.</p><p>Also, Swarm can also be integrated with Ansible which helps us to easily control our application stacks.</p><blockquote><strong>Good networking support:</strong></blockquote><p>Docker Swarm comes with very good networking solution with installation whereas in Kubernetes we need to integrated abstracted, third-party and loosely coupled networking solutions.</p><blockquote><strong>Load Balancing:</strong></blockquote><p>With Swarm, we can get automatic load balancing with DNS under the hood. But that is not the case with Kubernetes deployments.</p><p>However, we can use Nginx Ingress Controller with Kubernetes for load balancing purpose.</p><h4>Lightweight Kubernetes Solution — K3s</h4><p>As the main reason for choosing Docker Swarm over Kubernetes is because it is simple and lightweight.</p><p>However, Kubernetes too comes with its lightweight solution. Kubernetes K3s eliminates the heavy and complex architecture and provides a lightweight solution to Kubernetes.</p><p>K3s is packaged as a single binary (&lt;50MB) that implements the complete Kubernetes API and reduces the dependencies needed to install, run and manage a production Kubernetes cluster.</p><p>For Docker Swarm users, who want to switch to Kubernetes but are not sure if they are ready for full-on Kubernetes, K3s could be an ideal alternative to that.</p><h4>Final Thoughts</h4><p>Kubernetes is large and complex to operate and manage. One needs to have an in-depth understanding of the underlying concepts, abstract architecture and resource type usage for different scenarios. This makes learning Kubernetes relatively difficult and more confusing.</p><p>In other hand, Docker Swarm is simpler, easier to maintain, more familiar, and quicker to scale services.</p><p>We can use any tool for container orchestration depending on our need and complexity of the application. No matter which platform we choose, we will be well-positioned to scale and manage our containerized applications.</p><p>But, if we are looking for light-weight, minimalist and quicker container orchestration solution, Docker Swarm is the way.</p><p>That’s all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3186407f297a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NGINX —  The Right Way]]></title>
            <link>https://theankitojha.medium.com/nginx-the-right-way-d76a3c64d085?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d76a3c64d085</guid>
            <category><![CDATA[header]]></category>
            <category><![CDATA[hardening]]></category>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Thu, 14 Jul 2022 12:42:52 GMT</pubDate>
            <atom:updated>2022-07-14T12:42:52.194Z</atom:updated>
            <content:encoded><![CDATA[<h3>NGINX — The Right Way</h3><h3>Introduction</h3><p>Nginx (Engine-X) is the most popular open source web server. Nowadays, nginx is also used as a reverse proxy, HTTP cache, media streaming, load balancer and more.</p><p>It was originally created by a Igor Sysoev, a Russian engineer in October 2004. He created nginx as an attempt to solve the C10K Problem, which is a problem related to concurrently handling large number of connections “<strong>10K</strong>”.</p><p>Due to its robust ability to handle large number of concurrent connections, speed and many more, it is the web server of choice of various tech-giants like Google, Microsoft, Facebook, Apple, Twitter, Intel, Cisco, Adobe, VMWare and more.</p><h3>Best Security Practices</h3><p>We must secure our nginx web server properly before making it live. We must have a basic understanding of how nginx works and its configuration settings.</p><p>The main configuration file for nginx is <strong>nginx.conf </strong>which is located at <strong>/etc/nginx </strong>or <strong>/usr/local/etc/nginx </strong>depending on whether it is installed system-wide or for local user only.</p><p>We need to make some security checks in this main configuration file for best security practices. Also, for virtual hosts, we need to make changes to their configuration files, typically located inside the <strong>sites-enabled </strong>directory.</p><p>In this article, we are going through the best security practices that we can adopt while using NGINX.</p><h4>Controlling <a href="https://www.geeksforgeeks.org/buffer-overflow-attack-with-example/">Buffer Overflow Attacks</a></h4><p>We can minimize the risks of such attacks by setting the buffer size limitations for all clients in <strong>nginx.conf</strong> file.</p><p>Following directives can be used for this:</p><pre>client_body_buffer_size 1K;<br>client_header_buffer_size 1k;<br>client_max_body_size 1k;<br>large_client_header_buffers 2 1k;</pre><p><strong>where,</strong></p><p>client_body_buffer_size: client request body buffer size (default 8k or 16k)</p><p>client_header_buffer_size: client request header buffer size (1k is enough)</p><p>client_max_body_size: max accepted body size of client request</p><p>large_client_header_buffers: max number and size of buffers for large header</p><h4>Limit Available HTTP Methods</h4><p>There are various HTTP methods like GET, POST, HEAD and many more. If any of the methods is not required by the server, we must disable it from configuration.</p><p>This can be done by modifying <strong>nginx.conf </strong>file and adding the following lines under server block:</p><pre>if ($request_method !~ ^(GET|HEAD|POST)$ ) {<br>         return 405; # METHOD NOT ALLOWED<br>     }</pre><p>Or, we can add the following condition in the location block of the virtual host configuration file to filter out HTTP methods.</p><pre>location / {<br>limit_except GET HEAD POST { deny all; }<br>}</pre><p>In the above example, we can see that request method any other than GET, HEAD and POST will be returned with 405 message.</p><h4>Keep your software updated</h4><p>We must update the webserver from time to time to get latest updates, enhanced performance and security patches. Updating the system will do the work.</p><pre>sudo apt update &amp;&amp; sudo apt upgrade # Ubuntu<br>sudo yum update &amp;&amp; sudo yum upgrade # RHEL/CentOS</pre><h4>Use of SSL Certificates</h4><p>HTTP protocol is a plain text protocol and thus anyone can eavesdrop the information from in between. We must make use of HTTPS protocol which can be achieved by using SSL certificates.</p><p>SSL certificates helps to secure the content by cryptographically encrypting it using various algorithms. We can generate a free SSL certificate for our domain through Let’s Encrypt.</p><p>To setup the free SSL certificate using Let’s Encrypt, use the following link:</p><p><a href="https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/">https://www.nginx.com/blog/using-free-ssltls-certificates-from-lets-encrypt-with-nginx/</a></p><h4>Disable ‘<em>server_tokens’</em> directive</h4><p>This directive shows the version of the nginx webserver that is in use. This information is present in the HTTP response in server header.</p><p>If this is enabled, there might be a chance of information disclosure. Thus, we must disable this directive in <strong>nginx.conf</strong> file by setting:</p><pre>server_tokens off;</pre><h4>SSL and Cipher Suites Configuration</h4><p>Using older SSL/TLS protocol (v1, 1.1, 1.2) may lead to attacks such as <a href="https://www.acunetix.com/blog/web-security-zone/what-is-beast-attack/">BEAST attack.</a> Therefore, it is recommended to use newer version of TLS protocol by adding the following in <strong>nginx.conf</strong> file as:</p><pre>ssl_protocols TLSv1.2 TLSv1.3;</pre><p>Also, cipher suite configuration should also be done so that no vulnerable suites are supported. Cipher suites are the set of algorithms that help secure a network.</p><p>We can add the following directive that will let decide that the ciphers to be used are made on server-side and not on client-side.</p><pre>ssl_prefer_server_ciphers on;</pre><h4>Configure log files</h4><p>By default, all the server access and error logs are located inside <strong>/var/log/nginx</strong> directory as <strong>access.log</strong> and <strong>error.log</strong> files.</p><p>If we have to monitor the logs continuously for all virtual hosts, it is better to make a separate access and error log files for each host. This will help in discovering any error or attack attempt quickly and effectively.</p><p>We can do this by setting <strong>access_log</strong> and <strong>error_log</strong> directive in the vhosts file. They can be set within an <strong>http</strong>, <strong>server </strong>or <strong>location</strong> block.</p><pre>access_log /var/log/nginx/&lt;accessLogFileName&gt;.log<br>error_log  /var/log/nginx/&lt;errorLogFileName&gt;.log</pre><h4>Configuring security headers in NGINX</h4><p>Various security headers can be included in the nginx configuration file to secure the web server from various attacks. Some of the security headers are:</p><blockquote><strong>X-Frame Options</strong></blockquote><p>This response header is used to prevent <a href="https://en.wikipedia.org/wiki/Clickjacking">clickjacking attacks.</a> Basically, this header indicates whether the browser is allowed to render the page in &lt;frame&gt; or &lt;iframe&gt;.</p><pre>add_header X-Frame-Options &quot;SAMEORIGIN&quot;; </pre><p>Now, this will allow the browser to load the resources only from same origin.</p><blockquote><strong>X-Content-Type-Options</strong></blockquote><p>This header is used to protect against <a href="https://en.wikipedia.org/wiki/Content_sniffing">MIME sniffing.</a></p><pre>add_header X-Content-Type-Options &quot;nosniff&quot; always;</pre><blockquote><strong>Strict-Transport-Security</strong></blockquote><p>HTTP Strict Transport Security (HSTS) is a method to ensure that the website is only accessible using a secure HTTPS connection.</p><pre>add_header Strict-Transport-Security &quot;max-age=31536000; includeSubdomains; preload&quot;;</pre><p>The above header caches the information for the <strong>max-age</strong> period (typically 31536000 seconds ~ 1 Year). Also, the option <strong>includeSubdomains</strong> tells the browser that this policy is also applied to all of its subdomains too.</p><blockquote><strong>Content-Security-Policy</strong></blockquote><p>The CSP header enables us to control the resources which a user agent is allowed to load for a page.</p><pre>add_header Content-Security-Policy &quot;upgrade-insecure-requests;&quot;</pre><p><strong>upgrade-insecure-requests</strong> directive instructs to treat all insecure URLs of the website (HTTP) as though they are replaced over HTTPS. Various other directives can also be used with CSP.</p><blockquote><strong>X-XSS-Protection</strong></blockquote><p>This header helps to prevent against <a href="https://en.wikipedia.org/wiki/Cross-site_scripting">Cross-site Scripting Attacks</a> (XSS).</p><pre>add_header X-XSS-Protection &quot;1; mode=block&quot;;</pre><p>The NGINX server is now properly secured and ready to serve. However, there are many other security headers and directives in nginx to use according to the need.</p><p>Happy Learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d76a3c64d085" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux Server Hardening]]></title>
            <link>https://systemweakness.com/linux-server-hardening-fd6eece2178?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd6eece2178</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[vulnerability-management]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[server-hardening]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Wed, 22 Jun 2022 06:15:57 GMT</pubDate>
            <atom:updated>2022-07-19T11:23:24.575Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>The most secure server in the world is the one that is powered off and disconnected from the Internet. But if we want to use the server for IT operations, we must make sure that the server is secured properly before making it online.</p><p>Server Hardening is the process of making the system secure. Server hardening helps to address the best ways to minimize the points of vulnerability.</p><p>In this article, I am going to explain some of the best practices to secure the Linux servers.</p><p>Server Hardening can be done in various levels of the system:</p><ol><li><strong>Machine Level :</strong> Securing BIOS/drive encryption</li><li><strong>System Level:</strong> Password policies</li><li><strong>Network Level:</strong> Firewalls/Port configuration</li><li><strong>Logging and Auditing Level: </strong>alerting/logs/audits</li></ol><h3>Server Hardening Checklist</h3><p>Now, let’s look at some of the common and best practices that we can adopt to secure the system:</p><blockquote><strong>Regular System Update</strong></blockquote><p>This is one of the crucial step in system hardening because most of the flaws in the systems are because of some flaws in the software definition itself. Proper system update helps with reducing the risks related to it.</p><p>Depending on the distribution of Linux we use, we can update the system with the help of following commands:</p><pre>sudo apt update -y &amp;&amp; sudo apt upgrade -y         // For Ubuntu<br>sudo yum update -y &amp;&amp; sudo yum upgrade -y         // For RHEL/CentOS</pre><blockquote><strong>Block root login access from SSH</strong></blockquote><p>Since the root user of the system has administrative access, it should not be permitted for SSH login from remote desktops.</p><p>To disable the root access, make necessary changes to the SSH configuration file at <strong>/etc/ssh </strong>directory.</p><pre>cd /etc/ssh/<br>sudo vi sshd_config</pre><p>Edit the file and search for the pattern:</p><pre>#PermitRootLogin yes</pre><p>We need to uncomment the line and change the value to <strong>no</strong> in order to block the SSH root access.</p><pre>PermitRootLogin no</pre><p>Now, restart the SSH service to reflect the changes:</p><pre>sudo systemctl restart sshd.service</pre><blockquote><strong>Enforce strong password for root</strong></blockquote><p>Make sure to set a strong password for the root user. It is better to generate a hash for a plain text password and use the generated hash for the password.</p><p>This guarantees more security and reliability of the system.</p><blockquote><strong>Stop using default ports</strong></blockquote><p>Default ports are standard ports for any service or application. They are known to the world and anyone can enter into the system through the port if it is publicly accessible.</p><p>Therefore, we must set the random port for every service we use.</p><p><strong>For example:</strong></p><p>Instead of using port 22 for SSH, we can use any random port we wish.</p><p>Here’s how we can change the default SSH port ‘22’ to any other random port. SSH ports are well-known to the world and therefore it is very important that we set this to a random number.</p><p>Login as root user and edit the ssh configuration file as:</p><pre>sudo -i<br>cd /etc/ssh/<br>vi sshd_config</pre><p>Find the parameter ‘PORT 22’ on line 17 of the file (usually 17).</p><pre>PORT 22</pre><p>Change the value to the port number you wish to set for SSH and save the file.</p><p>Restart the SSH service with the help of following command and you are good to go.</p><pre>systemctl restart sshd.service</pre><p>The port for SSH has been changed successfully.</p><p>However, if the linux server is running on cloud, we need to add the new port in the Inbound Rules for the instance in its security group too.</p><blockquote><strong>Closed unused open ports</strong></blockquote><p>Open ports opens the attacking surface for hackers. Thus, we must not keep any unnecessary service or applications. And if we find one, we must disable the port or remove the application completely.</p><p>To check what ports are opened in the server, we can use the command:</p><pre>netstat -tulpn</pre><blockquote><strong>Keep things clean and sorted</strong></blockquote><p>We should never forget that <strong><em>Fewer packages = Fewer Vulnerabilities</em></strong></p><p>Remove any unnecessary package installed and never install any packages that is not needed. Application packages may contain vulnerabilities that can cause a serious damage to the system.</p><p>To remove any package installed, we can use the command:</p><pre>sudo apt-get remove &lt;package-name&gt; --purge        // Ubuntu</pre><pre>yum -y remove &lt;package-name&gt;                      // RHEL/CentOS</pre><p>Also make sure to clean up any old directories and files which are no longer in use.</p><blockquote><strong>Hardening Linux Kernel - /etc/sysctl.conf</strong></blockquote><p>In order to make a permanent modification, we need to configure the file <strong>sysctl.conf</strong> located inside the <strong>/etc/</strong> directory.</p><p>Login as a root user and edit the file as:</p><pre>sudo -i<br>vi /etc/sysctl.conf</pre><p>It can be used for configuring various security and performance tuning parameters such as:</p><h4>Control IP packet forwarding:</h4><p>Servers that acts as routers or gateways need to forward the packets and hence this feature needs to be enabled on such cases. However, in all other servers it is not needed and can be disabled by adding the following lines in <strong>sysctl.conf.</strong></p><pre>net.ipv4.ip_forward = 0</pre><h4>Source IP Address Verification</h4><p>Helps to check the spoofing attacks. Add the following lines to <strong>sysctl.conf</strong> to enable source IP address verification.</p><pre>net.ipv4.conf.all.rp_filter = 1</pre><h4><a href="https://en.wikipedia.org/wiki/Exec_Shield">Exec shield</a> Protection</h4><p>Exec shield is a security Linux kernel patch to avoid worms and other problems. Add the following lines to <strong>sysctl.conf</strong> to enable exec-shield protection.</p><pre>kernel.exec-shield = 1<br>kernel.randomize_va_space = 1</pre><h4>Prevention from <a href="https://en.wikipedia.org/wiki/Smurf_attack">smurf attacks</a></h4><p>Smurf attack refers to exploiting the IP broadcast address to create a DoS/DDoS attack through ICMP message packets.</p><p>We can stop smurf attacks by disabling the broadcast address as:</p><pre>net.ipv4.icmp_echo_ignore_broadcasts = 1</pre><h4><a href="https://en.wikipedia.org/wiki/SYN_flood">SYN Flood</a> Protection</h4><p>In SYN Flood attack, the system is flooded with a series of SYN packets.</p><p>Then, the system issues an SYN+ACK response and waits for the ACK (3 way handshake).</p><p>Since, the attacker never sends back the ACK, the entire resource of the system gets filled, known as backlog queue. And once the queue is full, it ignores the further requests from legit users for various services like http/mail etc.</p><p>To prevent the system from SYN Flood Attack, we need to enable the SYN cookies in <strong>sysctl.conf</strong> file by adding the following:</p><pre>net.ipv4.tcp_syncookies = 1</pre><blockquote><strong>Setup Login Banner</strong></blockquote><p>Setting up a banner does not contribute in securing the system.</p><p>However, banners can be used to show some messages or warnings when someone is establishing SSH connection to the server.</p><p>To enable the banner in SSH, first of all we need to create a <strong>banner file</strong> at <strong>/etc/mybanner. </strong>This will be a new file and fill it with desired message and save and exit from the file.</p><pre>sudo vi /etc/mybanner</pre><p>Now, we need to point this banner file by setting the file path to ssh configuration file. Search for the pattern ‘<strong>Banner</strong>’ and uncomment it and set the value to the file path of your banner file as:</p><pre>Banner /etc/mybanner</pre><p>Restart the SSH service:</p><pre>sudo service sshd restart</pre><p>Finally, let’s test our session by trying SSH login in our server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*OT8nEHuqO4Ud1Lv4NHT7yg.png" /></figure><p>It worked. The banner is displayed above the welcome message as shown above.</p><p>That’s all.</p><p>Happy Learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd6eece2178" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/linux-server-hardening-fd6eece2178">Linux Server Hardening</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch Backup and Restore with AWS S3 Bucket]]></title>
            <link>https://awstip.com/elasticsearch-backup-and-restore-with-aws-s3-bucket-9359e2e26504?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9359e2e26504</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[restore]]></category>
            <category><![CDATA[snapshot]]></category>
            <category><![CDATA[aws-s3]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Fri, 17 Jun 2022 08:48:45 GMT</pubDate>
            <atom:updated>2022-07-20T11:41:08.444Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*Hb0fbPge8u5ZCtMZvY3jzQ.jpeg" /></figure><p>Elasticsearch is an open source search and analytics engine based on Apache Lucene library. It provides extremely scalable search capabilities by taking unstructured data types from various sources and storing them in specialized formats for optimized language-based searches.</p><p>It also features a RESTful API design that gives developers incredible flexibility when calling different data formats for visualization and analysis.</p><p>In this article, I am going to explain how we can backup and restore the elastic search snapshots on AWS S3.</p><blockquote><strong>Create an S3 bucket and attach policy</strong></blockquote><p>Create an S3 bucket to hold all the index backups for the elastic-search server.</p><p>Goto: AWS Console &gt; S3 &gt; Create bucket</p><p>While creating bucket, make sure that the <strong>object ownership</strong> is given to the <strong>object writer</strong> itself and <strong>turn off </strong>the “<strong>Block Public Access</strong>” for the bucket for now.</p><p>We will later turn this on once we attach bucket policy to the bucket.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*AgKCtXzdwRK-8EkmafDuiQ.jpeg" /></figure><p>Now, its time to attach a policy in this bucket which will allow to list, post, get, delete the objects in the bucket.</p><p>Not to worry, once we attach policy and turn on the <strong>Block Public Access </strong>for the bucket, only authorized users can perform those actions and cannot be done publicly by anyone.</p><p><strong>Bucket Policy:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/101b874c42b4f3f18cd1a332dfe7f0b3/href">https://medium.com/media/101b874c42b4f3f18cd1a332dfe7f0b3/href</a></iframe><p>Once the policy is attached to the bucket, turn on the block public access for the bucket which will let the bucket to be accessed by authorized users only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*M-l6DfkcHBKWLaPj" /></figure><blockquote><strong>Create an IAM User and attach policy</strong></blockquote><p>Here, we are going to create an IAM user and attach certain policies to it to access the bucket.</p><p>Before creating an IAM user, let’s create a policy first.</p><p>Goto: AWS Console &gt; IAM &gt; Policies &gt; Create Policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l1uXXpWPCMuxjE66" /></figure><p><strong>IAM Policy:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5332e54557964541a1753df9930aa77/href">https://medium.com/media/b5332e54557964541a1753df9930aa77/href</a></iframe><p>Now, let’s create an IAM user and attach recently created policy to it.</p><p>Goto: AWS Console &gt; IAM &gt; Users &gt; Create User</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/0*eoVppESJJg5jgo9M" /></figure><p>Attach the policy to this IAM user and submit. Get the <strong>Access Key ID </strong>and <strong>Secret Key</strong> of the IAM user which will be needed in further steps.</p><blockquote><strong>Install necessary plugins and setup AWS access and secret keys</strong></blockquote><p>To use an S3 bucket as a snapshot repository, we need to install <strong>repository-s3 </strong>plugin on all of our es-nodes and setup <strong>AWS Access Key ID</strong> and <strong>Secret Key </strong>to the elastic-search keystore.</p><p><strong>If elastic-search is installed in server:</strong></p><p>Now, the installation of plugins if the elastic-search is installed in server can be done with the following command:</p><pre>cd /usr/share/elasticsearch/<br>sudo bin/elasticsearch-plugin install repository-s3</pre><p>After installing the plugin, restart the elastic-search service.</p><pre>sudo systemctl restart elasticsearch</pre><p>Now, we need to setup the access key and secret keys to the elastic-search keystore. This can be done with following commands:</p><pre>cd /usr/share/elasticsearch/<br>sudo bin/elasticsearch-keystore add s3.client.default.access_key<br>sudo bin/elasticsearch-keystore add s3.client.default.secret_key</pre><p>Insert the access and secret keys in respective places:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/0*wH0GEA7ZDTUiUq-j" /></figure><p><strong>If elastic-search is installed in docker:</strong></p><p>For this, we need to make changes to the <strong>Dockerfile </strong>for elastic-search with the following parameters:</p><pre>RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3</pre><pre>RUN echo &quot;&lt;your_access_key&gt;&quot; | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key</pre><pre>RUN echo &quot;&lt;your_secret_key&gt;&quot; | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key</pre><p>Make sure to replace &lt;your_access_key&gt; and &lt;your_secret_key&gt; with your IAM user’s access and secret keys and spin up your ELK stack in docker.</p><blockquote><strong>Register a repository</strong></blockquote><p>Goto: Kibana Dashboard &gt; Settings &gt; Snapshots and Restore &gt; Repositories</p><p>Register a new repository with repository type set to S3 and click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aDVxh0lkjdRDwTZ7" /></figure><p>Enter the bucket name which you want to use as a repository and leave everything else as default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-E2_DFhDnxZBkg44" /></figure><p>After registering the repository, click on the repository name and test whether it is connect to the elastic-search server or not.</p><blockquote><strong>Create a snapshot</strong></blockquote><p>To create a snapshot, first of all we need to<strong> create a policy.</strong></p><p>Go to policies and create a new one. Set the policy name and snapshot name of your choice, set the cron expression for automatic snapshots and click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SK1u5E8WEaOYQM_u" /></figure><p>Set other configurations if needed, else leave all to default. Here, I had made some changes to snapshot retention policy as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mnChXhUFfUznBbIe" /></figure><p>After creating a policy, navigate to the <strong>Policies </strong>tab and click on <strong>Run </strong>(play button icon) to trigger the snapshot process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P7R0IQ6wsH3nDF59" /></figure><p>As soon as it stops running, navigate to <strong>Snapshots </strong>tab, where we can see our newly created snapshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lf1QO-nJLQGDr9jc" /></figure><p>And, if we look into our bucket in aws console, we can see that the snapshot is created successfully in our bucket repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3bdiCykp9qDbLlEj" /></figure><p>This is how we can take snapshot of our elastic search server and upload it to s3 bucket.</p><h3>Restore from the snapshot</h3><p>Now to restore from the snapshot to any other elastic search server, we need to go through the following steps:</p><blockquote><strong>Install necessary plugins and setup AWS access and secret keys</strong></blockquote><p>Again, we need to install the <strong>repository-s3 </strong>plugin and setup the access and secret keys in the elastic-search server where we want to restore our index.</p><p>This step is same as we did before.</p><p><strong>If elastic-search is in server:</strong></p><p>Now, the installation of plugins if the elastic-search is installed in server can be done with the following command:</p><pre>cd /usr/share/elasticsearch/<br>sudo bin/elasticsearch-plugin install repository-s3</pre><p>After installing the plugin, restart the elastic-search service.</p><pre>sudo systemctl restart elasticsearch</pre><p>Now, we need to setup the access key and secret keys to the elastic-search keystore. This can be done with following commands:</p><pre>cd /usr/share/elasticsearch/<br>sudo bin/elasticsearch-keystore add s3.client.default.access_key<br>sudo bin/elasticsearch-keystore add s3.client.default.secret_key</pre><p><strong>If elastic-search is in docker:</strong></p><p>For this, we need to make changes to the <strong>Dockerfile </strong>for elastic-search with the following parameters:</p><pre>RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3</pre><pre>RUN echo &quot;&lt;your_access_key&gt;&quot; | bin/elasticsearch-keystore add --stdin --force s3.client.default.access_key</pre><pre>RUN echo &quot;&lt;your_secret_key&gt;&quot; | bin/elasticsearch-keystore add --stdin --force s3.client.default.secret_key</pre><p>Make sure to replace &lt;your_access_key&gt; and &lt;your_secret_key&gt; with your IAM user’s access and secret keys and spin up your ELK stack in docker.</p><blockquote><strong>Register a repository</strong></blockquote><p>This step is also same as we did before. The only difference is that now we are doing it in destination elastic server where we want to restore the snapshot.</p><p>Goto: Kibana Dashboard &gt; Settings &gt; Snapshots and Restore &gt; Repositories</p><p>Register a new repository with repository type set to S3. Also, set the bucket name from where we want to restore the snapshots.</p><p>Now, verify whether your repository is connected to elastic search nodes properly or not.</p><blockquote><strong>Restore snapshot</strong></blockquote><p>Finally, we are going to restore the snapshot in our new elastic search server.</p><p>Goto: Kibana Dashboard &gt; Settings &gt; Snapshot and Restore &gt; Snapshots</p><p>Here, we can see the snapshots that has been taken in this repository.</p><p>To restore any particular snapshot, click on the <strong>Download </strong>like icon under ‘Actions’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ktIGxAT4qTq9RtFV85V4Qg.png" /></figure><p>While restoring the snapshots, make sure to unselect the option to restore <strong>all indices, including system indices</strong>.</p><p>The system indices are already created by default and there’s no need to import them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OtWHDaGBzFaCf5DVS9GYw.png" /><figcaption>System indices starts with a dot (.)</figcaption></figure><p>Set other configurations if needed, else leave all to defaults and click Next and proceed to <strong>Restore Snapshot.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*8tCmKePnrwzII8nSG42wqg.png" /></figure><p>That’s all. We successfully restored the snapshot in our new elastic search server.</p><p>Thanks.</p><p>Happy Learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9359e2e26504" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/elasticsearch-backup-and-restore-with-aws-s3-bucket-9359e2e26504">Elasticsearch Backup and Restore with AWS S3 Bucket</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS S3 Bucket Migration between different AWS Accounts]]></title>
            <link>https://awstip.com/aws-s3-bucket-migration-between-different-aws-accounts-15c7dba0454a?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/15c7dba0454a</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[iam-user]]></category>
            <category><![CDATA[bucket-migration]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Thu, 16 Jun 2022 05:09:23 GMT</pubDate>
            <atom:updated>2022-07-21T10:58:17.093Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*uZYb03EC5s9LFks8G1DsaQ.jpeg" /></figure><p><strong>Amazon S3 </strong>stands for <strong>Simple Storage Service </strong>which provides object storage service. A user can interact with S3 bucket via different modes like console, CLI or APIs.</p><p>It is an object based storage method, meaning we can store anything like images, files, pdfs, videos and many more to s3 bucket.</p><p>In this article, I am going to explain how to migrate the objects of S3 bucket from source to destination AWS account.</p><h3><strong>Steps:</strong></h3><blockquote><strong>Get the Account ID for the destination account</strong></blockquote><p>First of all, we need to go to the console of the destination account and note down its Account Id.</p><p>Goto: AWS Console &gt; Click on top right corner</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqukTD4aF2jUyHj86e7OHg.jpeg" /><figcaption>AWS Account ID details</figcaption></figure><blockquote><strong>Bucket policy Attach — Source Bucket</strong></blockquote><p>Goto: AWS Console &gt; S3 &gt; sourceBucket &gt; Permissions &gt; Bucket Policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_FarRmSBUiDVGsrGnNhcg.jpeg" /></figure><p>Now, we need to attach the following bucket policy to the source bucket which is to be migrated.</p><p><strong>Bucket Policy:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5348e79834d7f6664c79f5e2bd4725f/href">https://medium.com/media/b5348e79834d7f6664c79f5e2bd4725f/href</a></iframe><p>Make sure to replace the values with your real destination AWS account number and source s3 bucket names.</p><blockquote><strong>IAM Policy Attach — Destination IAM User</strong></blockquote><p>Goto: AWS Console &gt; IAM &gt; Policies &gt; Create Policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AbkcgTducP1V3i-TcOv2JQ.jpeg" /></figure><p>Copy the following and create a new IAM policy.</p><p><strong>IAM Policy</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/570a54b3e605b9cdfab658acecfa1912/href">https://medium.com/media/570a54b3e605b9cdfab658acecfa1912/href</a></iframe><p>Now, we need to create a new user and attach the IAM policy to the newly created user in the destination account.</p><p>Goto: AWS Console &gt; Users &gt; Add User</p><p>While creating a user, we are meant to setup necessary permissions for the user. During this step, attach the recently created IAM policy to this user. This will grant the user with permission to do the migration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mRpw_QaJW2m05DY4iJyO1g.jpeg" /></figure><blockquote><strong>Installing AWS CLI and configuring Access Key ID and Secret Access Key</strong></blockquote><p>After completing the above steps, now we need to install the AWS CLI in our machine with the help of following commands:</p><pre>sudo apt install awscli // For Ubuntu/Debian</pre><pre>sudo yum install awscli // For RHEL/CentOS</pre><p>Once the <strong>awscli</strong> is installed, now we need to setup the access key ID and secret key ID of the IAM user that we created in destination account.</p><p>For this, type the command:</p><pre>aws configure</pre><p>We will be asked for the <strong>Access Key ID </strong>and <strong>Secret Access Key. </strong>Fill in the respective fields with your access key, secret key and press enter.</p><p>Now, we are ready for migration process.</p><blockquote><strong>Migrate/Copy s3 bucket objects to destination bucket</strong></blockquote><p>Finally, we can migrate the objects in s3 bucket with the help of following command:</p><pre>aws s3 sync s3://YOUR-SOURCE-BUCKET-NAME-HERE s3://YOUR-DESTINATION-BUCKET-NAME-HERE --source-region SOURCE-REGION-NAME --region DESTINATION-REGION-NAME</pre><p>Make sure to replace the bucket names and regions for both source and destination.</p><p>That’s all.</p><p>Happy Learning :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15c7dba0454a" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-s3-bucket-migration-between-different-aws-accounts-15c7dba0454a">AWS S3 Bucket Migration between different AWS Accounts</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Static Malware Analysis: Identifying Malware]]></title>
            <link>https://theankitojha.medium.com/static-malware-analysis-identifying-malware-c8c7060799?source=rss-5c8f459a424d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8c7060799</guid>
            <category><![CDATA[malware-analysis]]></category>
            <category><![CDATA[static-analysis]]></category>
            <category><![CDATA[malware]]></category>
            <category><![CDATA[static-analysis-tool]]></category>
            <dc:creator><![CDATA[Ankit Ojha]]></dc:creator>
            <pubDate>Fri, 19 Jun 2020 06:06:01 GMT</pubDate>
            <atom:updated>2020-06-19T06:06:01.085Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Introduction</strong></blockquote><p>As soon as you get the malware, you will always try to jump right in and try to tear it apart and get the information from it using various techniques. However, there are few things you should make sure you do before performing any analysis, as these will help the speed and guide everything else you do.</p><p>One of the techniques is: <strong>File Identification</strong></p><p>Malware authors constantly try to trick users into opening up their malware, and one of the ways they do this is by disguising a file to look like it’s safe.</p><p>So, as a malware analyst, one of the first things we have to do is figure out what type of file we are dealing with. They must apply various techniques that will help them to identify exactly what they’re examining.</p><blockquote><strong>Static Analysis</strong></blockquote><p>As we know that, static analysis techniques examine the characteristics of malware without executing it. In this phase, various pieces of information about the malware are gathered together which can be used in the dynamic analysis phase when we execute it and see what it really does.</p><p>In other words, the data that we gathered in this static analysis gives glimpses into what the malware can do. And, these can be confirmed during the dynamic analysis phase.</p><blockquote><strong>Focus Your Analysis</strong></blockquote><p>In static analysis, we need to focus on various things like:</p><p>1. You may not initially know the type of the file, but it is the first thing you must figure out.</p><p>2. Has anyone already analyzed this malware? Knowing this can save you a lot of time.</p><p>3. The embedded strings on the malware can reveal a lot of information on what the malware can do.</p><p>4. The Windows PE executable header reveals a lot of information about the malware. It can be used as our advantage to help us figure out how the malware will behave.</p><p>5. We know that the packers compress a program and hide its internals from us. So, if we can determine if the malware is packed and what packer was used, we can unpack it and get the underlying information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*XNCcplv35WtpoOjw_IlUcg.png" /></figure><h3><strong>NOTE:</strong></h3><p>Despite the fact that the static analysis gets its information without executing the malware, doesn’t mean that it’s any less dangerous. We still need to be very careful during static analysis, and only analyze the malware in Sandbox. This is because there are many tools that may open or execute a file in order to obtain the information you want, including static analysis tools.</p><p>We must also be very aware of the tools we use. There are many tools that are available for analyzing malware. However, it is recommended to treat any tool like malware and analyze them. This is basically for two reasons:</p><ol><li>This will allow you to see how the tools work. They may modify the registry entries, drop files, or setup services. Knowing this will help you exclude the activity later on when you run the tool, especially during the dynamic analysis.</li><li>Unfortunately, some tools are located in unscrupulous places on the internet. Some of the malware analysis tools are backdoors or malware themselves. Treating these tools like malware will allow seeing any odd modifications to your Sandbox or weird connections to websites caused by these tools. And, in case any such suspicious activity is detected, we must avoid using that tool.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*FzxxKvjGxMID-IsqYTpjWg.png" /></figure><blockquote><strong>File Identification</strong></blockquote><p>We know that we may not know the type of file we are dealing with at the start of our malware analysis. This is because the attackers can do everything they can to trick the users into opening up the malware by disguising their files.</p><p>Of course, some of the things they do to accomplish this always makes analysis a little bit more difficult.</p><p>One of the most common ways the attackers do this is by using <strong>“double extensions.” </strong>Just like it sounds, an attacker adds an extension that looks like “.doc” before the real extension. This attempts to trick the user into thinking that the file is not executable when it, in fact, is.</p><p>Another common method attacker gets around signature-based analysis is by packaging the malware in self-extracting archives. There are a lot of archiving utilities, and many of them, such as NSIS and WinRAR, have their own scripting languages. Attackers will package their malware in these archives, including a script to install it and send it off. And, when the archive is executed, the internal script extracts the malware and executes it.</p><p>You can think of this as a poor man’s packer.</p><p>Speaking of Packers, many file identification tools will also examine executables for the indication of packers. Some of them will also give hints on how to unpack the malware.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*f5FLQMXzBNi1pF-NCWiAzg.png" /></figure><blockquote><strong>File Signatures</strong></blockquote><p>Many file types, such as executables and documents, begin or end with specific bytes known as magic bytes. File identification tools use these bytes as signatures in order to identify the type of file they are examining.<br> <strong>e.g.</strong> this is a screenshot of the Windows PE program, the format most Windows executables are in. There are a number of signatures within the file that can be used to identify it as such.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*6BXbQ3aHVgkEU6YVzWrK9w.png" /></figure><p>Here, the first two bytes of the file are MZ. All windows programs will have MZ as their first two bytes.</p><p>In another hand, most of the PE programs will have statements such as “This program cannot be run in DOS mode” down here. However, this is not necessary. And, some packers might change or remove this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*htMU4YbtcPL4kT46wkpVUA.png" /></figure><p>Finally, at offset HEX E0 is the start of the PE header with the two bytes, PE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*qTliQ_YDCyjU9gV1GTUIbw.png" /></figure><p>Well, you probably won’t have to open up a hex-editor to manually identify the files. It is not a bad idea to know the signatures of a number of common file types that you might come across.</p><p>Here are some common file signatures you should know:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*iBPT6_24VkbK5LJZUFz1_A.png" /></figure><blockquote><strong>File Identification Tools</strong></blockquote><p>There are a number of tools available to provide file detection.</p><p><strong>File:</strong></p><p>· One of the most commonly used file identification tools.</p><p>· It works by comparing the file header with known signatures of different file types, which is stored in a plain text file called the magic file.</p><p>· This tool is fast, simple, and effective, but it does not provide much in-depth information on the file you are looking at.</p><p>· They usually come standard on most UNIX-based systems, including Linux and macOS X. Windows has a version too for this tool.</p><p>Here is an example of “File” being run on an executable named <strong>openme.doc.exe</strong>. Notice the double extension used by the attacker to try and disguise the file type. Now, <strong>File </strong>tells us that it is a PE32 GUI executable, simple and easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*51B4xrebqloSq4qNiMPIfA.png" /></figure><p><strong>Exeinfo PE:</strong></p><p>· This tool works only on Windows executables, but it returns a lot of information on them, including possibly what compiler was used to create it.</p><p>· This tool is great at detecting packers. If the file you give it is packed, there’s a good chance Exeinfo PE will be able to tell you what packer it is.</p><p>· Even better, Exeinfo will give you hints on how to unpack it. This is really great as we don’t have to hunt all over the internet to figure out how to get at the underlying information in the file.</p><p>· It is a GUI application but also comes with a command-line version in case you want to use it in automation scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*8h3uPIZP3d2EIMAZ0RjSsw.png" /></figure><p>Here we are running Exeinfo PE on <strong>openme.doc.exe</strong>.</p><p>There is a lot of information packed into this tiny interface, but the part we want to look at is at the bottom. The section in dark blue describes what it found in the executable.</p><p>Earlier, <strong>File</strong> has told us that it is a Windows executable, Exeinfo PE tells us we have a 64-bit executable which has been packed with a packer called UPX.</p><p>Just below the type information is the unpacking hint (in the light blue section). This tells us how we can unpack the UPX packed program.</p><p>UPX is actually a very commonly used packer. It’s open-source and available for most systems, so it’s not uncommon to find malware packed with it. Fortunately, unlike most packers, the same UPX program that packs executables can also unpack them. UPX can be downloaded at <a href="http://upx.sf.net/">http://upx.sf.net</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*5qFhr986wx6-IXijIYoatQ.png" /></figure><p><strong>TrID:</strong></p><p>· Available for both Windows and Linux.</p><p>· Unlike the last two programs, which use the signatures for file type detection, TrID uses a pattern database to determine the file type.</p><p>· If the file contains any of the patterns TrID has, it displays them.</p><p>· TrID will also show you how likely the file has each of the patterns detected. This is very helpful, as the choices TrID gives is sometimes more accurate than the other signature-based tools.</p><p>· The pattern database is updated fairly regularly, so it’s always able to detect new types of files.</p><p>· TrID is a command-line tool, and the only required option you have to give it is the path to the pattern database. You can do this through the -d: option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*YUVaUA0gxgi3ApILU5lO4g.png" /></figure><p>Finally, the TrID is run against <strong>openme.doc.exe</strong>.</p><p>In this case, TrID tells us that there is an 87.1% chance we have a UPX packed program, with a 6.4% chance we just have a regular Windows or DOS executables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*qydV2cn8pol3ZFFSKYjihg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8c7060799" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>